diff --git a/AGENTS.md b/AGENTS.md index 753aba9a77..49d1a1b368 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -80,7 +80,7 @@ Use these before guessing. All are off unless the variable is set; pass them as | A newly lowered program changes the image | `EDEN_NX2_REJECT_HASH=,...` (native program keys from shader dump file names, or shader hashes from the frame dump) | Those programs are rejected as if unsupported; the log shows `rejected by EDEN_NX2_REJECT_HASH` per hit. Bisect by halving the list. The key covers the captured code, so recompute it when the capture rules change. | | Draw and render-state questions | `EDEN_NX2_RENDER_TRACE=1`, filter `*:Info HW.GPU:Debug Render.Vulkan:Debug Shader:Debug` | Per-draw state, rejected stages and zero-format texture descriptors. | | Video playback | `EDEN_NX2_VDEC_TRACE=1`, `EDEN_NX2_VDEC_DUMP=` | Decoder submit, decode and composite trace; decoded frames. | -| Performance and frame pacing | `EDEN_NX2_GPU_PROFILE=1`, `EDEN_DKB_FRAME_TRACE=1` | GPU thread, back thread and Vulkan worker time per second by zone, GPU waits by the zone they happen in, waits for the back by site, and spin mutex contention; per-present interval, CPU and fence timing. The profile also logs `Async read-back per second` every 10 s: guest reads of GPU-written memory, their waits, and macro DMA reads that waited for the split's back or drained it. | +| Performance and frame pacing | `EDEN_NX2_GPU_PROFILE=1`, `EDEN_DKB_FRAME_TRACE=1` | GPU thread, back thread and Vulkan worker time per second by zone, GPU waits by the zone they happen in, waits for the back by site, and spin mutex contention; per-present interval, CPU and fence timing. The #805 integration also logs `GPU waits in ... s` every 10 s with guest reads, reactive flushes and front/back waits; it retains #805's read tracing rather than the port's duplicate async-readback counters. | | HostTiming CPU or timer lateness (late guest timers, audio or vsync jitter) | `EDEN_NX2_HOSTTIMING_STATS=1`, or live with the debug server's `host_timing {"stats": true}` | One `HostTiming wait` line per second: waits, waits woken by newly scheduled events, OS sleep and `WaitFor` time (ms/s), how late timed waits ended and how late events fired (p50/p99/max µs), and the thread's CPU %. | | Shader stutter or not, boot and cache costs | `EDEN_NX2_STUTTER_TRACE=1` | One `Nx2Stutter` line per frame presented over 33 ms after the last, with its top cause (gfx/compute pipeline built on the GPU thread, pipeline wait, shader capture, texture, buffer, query/fence, present, other) and each GPU thread's ms; `Nx2StutterSummary` every 10 s (frame-time histogram, builds, cache lookups, cache file size); `Nx2Boot` boot phases with commit and peak commit. Frames over `EDEN_NX2_STUTTER_DETAIL_MS` (default 50) also get `Nx2StutterDetail`: each GPU thread's eight largest zones (ms/calls) and the rare sites that ran (device memory allocate/free, staging, buffer and image create/delete/GC, synchronous downloads, `Scheduler::Finish`), with each run over 2 ms by thread; the summary adds the sites per 10 s. The category totals in the summary are every frame's steady cost, not hitch time. `python3 -I tools/nx2/cache_metrics.py parse ` turns a log into JSON, `compare` A/B runs; `tools/nx2/cache_bench.py` runs the cold/warm matrix on the PC. | | Command corruption or pop-in | `EDEN_NX2_PB_CHECK=1`; A/B with `EDEN_NX2_PB_SNAPSHOT=0` or `EDEN_NX2_FETCH_DRAIN=1` | How many pushbuffer entries change between submit and parse. | diff --git a/NX2_GPU_THREAD_SPLIT.md b/NX2_GPU_THREAD_SPLIT.md index 75fd0e4105..83db7c3213 100644 --- a/NX2_GPU_THREAD_SPLIT.md +++ b/NX2_GPU_THREAD_SPLIT.md @@ -61,3 +61,80 @@ with stage 2, 50,000-66,500 with stage 3. The front waits ~4,000-5,000 times a s acquires, MME DMA reads); the back still spends ~200 ms a second outside its zones (ring dispatch, method sink). Clean A/B (NX2_NOTES row 145): 1.46x throughput, frames 17% shorter; on by default since. The back is ~92% busy, ~53% of it draw setup: the next step is there, not the ring. + +## Fast draw ports on the #805 performance branch (2026-11-09) + +The port of nx2 `1c25fbdfb4` (#921) makes full resolves and texture memo hits use +the same image-preparation predicate. Empty descriptor views backed by the null +image are excluded even when their view id is not the null view id. Delta draws +fall back when a sampled image needs an upload or alias synchronization, or when +the draw's serial, mappings, command buffer or render pass changed before the +caches were locked. Rescaling ends the chain too. + +Only the uniform-keep and image-descriptor replay portions of nx2 `a4f6287327` +(#949) are ported: `6b38851501`, `ee98215f14`, and the CPU-write follow-up +`3e86407514`. The test-runtime support in `b2375cff7f` already exists on #805. +Its semaphore, waiter and acquire/drain replacements are not included; #805 +retains its own implementations. The subsequent nx2 merge keeps the independent +scheduler-owner guard on buffer-cache writes from CPU/service threads, without +replacing #805's back-owned `Run` operations with nx2's `back_quiet` drain scheme. + +Small streamed uniforms can reuse the previous allocation without rereading +guest memory when the stage, slot, device range and command-buffer tick still +match, and no overlapping inline data, notified CPU write, GPU write or buffer +deletion invalidated it. More than eight notified writes between draws disable +keeps for the next draw. Unsynchronized CPU writes unknown to the caches remain +an assumption to check on the game. Per-stage image descriptor entries and +rescaling words are replayed until the views are recaptured or the scale epoch +changes; buffer descriptors still bind on every draw. + +Validate a fixed scene with these diagnostics separately (set each to `1`): + +- `EDEN_NX2_TEXTURE_MEMO_VERIFY`: compare views, samplers and prepared image sets. +- `EDEN_NX2_FAST_DRAW_VERIFY`: full-resolve comparison; this deliberately bypasses + delta draws, so it does not exercise uniform keeps or descriptor replay. +- `EDEN_NX2_UBO_KEEP_VERIFY`: reread would-be kept uniforms and report changed bytes. +- `EDEN_NX2_DELTA_DESCRIPTORS_VERIFY`: regenerate and compare image descriptors + and rescaling words. +- `EDEN_NX2_GPU_PROFILE`: report keep/replay counts and draw fallback reasons; + value `2` also retains #805's fine uniform-binding counters. + +After building the test target, run `tests.exe "[buffer_ports]"`, +`tests.exe "[video_core][texture_cache]"`, and +`tests.exe "Uniform stream*"`. These ports were reviewed without building on +the porting machine; compilation, runtime verification and visual A/B testing +are still required. #949's reported Bananza gain (+8–9%, back thread about +56.0 to 52.2 ms/frame on a Ryzen 9 5900X) includes its acquire changes and does +not predict the gain of this partial port in Ocarina of Time. + +## nx2 integration through 1a5e995a38 + +- #1028 does not replace #805's read downloads, write tracking, sampled method + profiling or diagnostics. Its correctness fixes are retained: discard/DMA + destinations first land overlapping read downloads, and older deferred query + results cannot mark newer GPU writes clean. Synced #874 payload callbacks still + do not advance `fenced_landed` or retire an older deferred report's tracking; + `FencedPayloadLanding` remains authoritative. +- #979/#1014 retain every registered semaphore region and the direct-release + shadow. Mirrored releases are emitted only when `ViewApplies()` is false + (unsplit, back macros, or delayed/high-accuracy releases). The full #805 front + view continues to decide from the original releases, never a second mirrored + counter. Its unbounded, source-tracking waiter collection is retained. +- Shader cache v2 replaces the legacy Vulkan cache. #970's no-flush TIC reads + apply to worker translation too, so no pool-wide flush is introduced before + launching workers. As with #970, pending GPU writes to descriptors remain a + runtime validation risk. AMD shader exceptions, the image recycler and memory + usage caching are unchanged. +- #965/#1039 add stutter sites, GPU waits by zone, queue/present waits, contention + and opt-in live-read comparisons around the existing split. Method profiling + still takes the fast path and samples timing, rather than timing every write. +- #1056's macro-on-back port arrived during integration. Keep #805's full effect + analysis and back-macro machinery, but take the review fixes: recheck command + pages even when the writer wait did not block, and remember the actual last + writer at each host sync rather than waiting through unrelated intervening work. + +In-game validation should cover low/default and high/delayed fence modes, +cross-channel semaphore waits, concurrent CPU read-back, cold/migrated/warm v2 +caches, and the fast-draw checks above. The integration build disables +`BUILD_TESTING`; the updated C++ regression sources must be run separately in a +test-enabled build. diff --git a/src/common/nx2_sampler.cpp b/src/common/nx2_sampler.cpp index 244543709e..31c1ffabe4 100644 --- a/src/common/nx2_sampler.cpp +++ b/src/common/nx2_sampler.cpp @@ -34,6 +34,7 @@ constexpr size_t MaxDepth = 40; struct Sample { unsigned thread; unsigned depth; + bool faulted; ///< the walk stopped at an access violation std::array frames; }; @@ -146,15 +147,15 @@ std::vector FindThreads(const std::vector& patterns) { // could be paused holding the heap lock. A wrong unwind step (code with missing or wrong unwind // data) leaves a garbage stack pointer: reading through it crashed the process, so the walk // stops once the stack pointer leaves the thread's stack. -unsigned Walk(CONTEXT context, std::array& frames, const NT_TIB* tib) { +void Walk(CONTEXT context, std::array& frames, const NT_TIB* tib, + unsigned& depth) { if (tib == nullptr) { - return 0; + return; } const auto on_stack = [tib](DWORD64 rsp) { return rsp >= reinterpret_cast(tib->StackLimit) && rsp + 8 <= reinterpret_cast(tib->StackBase) && (rsp & 7) == 0; }; - unsigned depth = 0; while (depth < MaxDepth && context.Rip != 0 && on_stack(context.Rsp)) { frames[depth++] = context.Rip; DWORD64 image_base{}; @@ -174,6 +175,26 @@ unsigned Walk(CONTEXT context, std::array& frames, const NT_T RtlVirtualUnwind(UNW_FLAG_NHANDLER, image_base, context.Rip, function, &context, &handler_data, &establisher_frame, nullptr); } +} + +// RtlVirtualUnwind also reads through the frame register, which the stack check above doesn't +// cover: a thread paused in a fiber switch with Rbp 0 made it read address 0, and the access +// violation took the process down (EDEN_NX2_SAMPLE=CPUCore_*). Such a walk keeps the frames it +// found. +unsigned GuardedWalk(const CONTEXT& context, std::array& frames, + const NT_TIB* tib, bool& faulted) { + unsigned depth = 0; + faulted = false; +#ifdef _MSC_VER + __try { + Walk(context, frames, tib, depth); + } __except (GetExceptionCode() == EXCEPTION_ACCESS_VIOLATION ? EXCEPTION_EXECUTE_HANDLER + : EXCEPTION_CONTINUE_SEARCH) { + faulted = true; + } +#else + Walk(context, frames, tib, depth); +#endif return depth; } @@ -257,8 +278,15 @@ std::string Report(const std::vector& targets, const std::vector std::map inclusive; std::map self_caller; unsigned total = 0; + unsigned faulted = 0; + unsigned empty = 0; for (const Sample& sample : samples) { - if (sample.thread != thread || sample.depth == 0) { + if (sample.thread != thread) { + continue; + } + faulted += sample.faulted ? 1 : 0; + if (sample.depth == 0) { + ++empty; continue; } ++total; @@ -289,7 +317,9 @@ std::string Report(const std::vector& targets, const std::vector sorted[i].second.c_str()); } }; - text += fmt::format("=== thread \"{}\": {} samples\n", targets[thread].name, total); + text += fmt::format("=== thread \"{}\": {} samples ({} without a stack, {} walks stopped " + "by a fault)\n", + targets[thread].name, total, empty, faulted); print("self (where the thread is)", self, 60); // The hottest instructions as module offsets, for llvm-symbolizer --relative-address // (which line of a function, inlined code included). @@ -371,7 +401,8 @@ std::string SampleThreads(const std::vector& patterns, double secon if (GetThreadContext(handle, &context)) { Sample& sample = samples.emplace_back(); sample.thread = i; - sample.depth = Walk(context, sample.frames, targets[i].tib); + sample.depth = + GuardedWalk(context, sample.frames, targets[i].tib, sample.faulted); } ResumeThread(handle); } diff --git a/src/core/hle/service/nx2_gsv.h b/src/core/hle/service/nx2_gsv.h index a21b4eeee9..9efbd35e34 100644 --- a/src/core/hle/service/nx2_gsv.h +++ b/src/core/hle/service/nx2_gsv.h @@ -39,6 +39,7 @@ #include "core/frontend/framebuffer_layout.h" #include "core/perf_stats.h" #include "video_core/gpu.h" +#include "video_core/gpu_wait_stats.h" #include "video_core/framebuffer_config.h" #include "core/core_timing.h" #include "core/hle/kernel/k_client_session.h" @@ -1802,6 +1803,7 @@ class INx2Vi2Session final : public ServiceFramework { m_vsync->trace_present.store(m_presents, std::memory_order_relaxed); } const u64 present = Core::g_nx2_present_count.fetch_add(1, std::memory_order_relaxed) + 1; + VideoCommon::GpuWaitStats::NotePresent(); if (Core::g_nx2_present_time_services.load(std::memory_order_relaxed) != 0) { const u64 present_ns = std::chrono::duration_cast( std::chrono::steady_clock::now().time_since_epoch()) diff --git a/src/core/memory.cpp b/src/core/memory.cpp index c55d716434..7344e75c83 100644 --- a/src/core/memory.cpp +++ b/src/core/memory.cpp @@ -35,6 +35,7 @@ #include "core/hle/kernel/k_process.h" #include "core/memory.h" #include "video_core/gpu.h" +#include "video_core/gpu_wait_stats.h" #include "video_core/host1x/gpu_device_memory_manager.h" #include "video_core/host1x/host1x.h" #include "video_core/rasterizer_download_area.h" @@ -891,6 +892,7 @@ struct Memory::Impl { return; } const size_t core = system.GetCurrentHostThreadID(); + VideoCommon::GpuWaitStats::CountRasterizerRead(core); if (core >= Core::Hardware::NUM_CPU_CORES) { // A service thread has no slot in the arrays below, and it does not use a slot of a // core. It keeps no area, so it asks on each read. @@ -943,6 +945,7 @@ struct Memory::Impl { sys_core_guard.unlock(); } }; + VideoCommon::GpuWaitStats::CountRasterizerWrite(core); gpu_device_memory->ApplyOpOnPointer(p, scratch_buffers[core], [&](DAddr address) { auto& current_area = rasterizer_write_areas[core]; PAddr subaddress = address >> YUZU_PAGEBITS; @@ -952,8 +955,11 @@ struct Memory::Impl { // we should let the compiler just do it's thing... if (current_area.last_address != subaddress) { // Short circuit the need to check for address/size + const u64 check_start = VideoCommon::GpuWaitStats::Now(); auto const do_collection = (address != 0 && size != 0) && system.GPU().OnCPUWrite(address, size); + VideoCommon::GpuWaitStats::AddWriteCheck(core, VideoCommon::GpuWaitStats::Now() - + check_start); if (do_collection) { current_area.last_address = subaddress; } else { diff --git a/src/tests/CMakeLists.txt b/src/tests/CMakeLists.txt index 1ba28c25f8..1b2962add4 100644 --- a/src/tests/CMakeLists.txt +++ b/src/tests/CMakeLists.txt @@ -191,6 +191,7 @@ add_executable(tests video_core/turing_macro_effects.cpp video_core/turing_macro_ram.cpp video_core/uniform_stream_reuse.cpp + video_core/upload_order.cpp video_core/usage_tracker.cpp video_core/shader_invalidation.cpp video_core/descriptor_table.cpp diff --git a/src/tests/video_core/buffer_cache_ports.cpp b/src/tests/video_core/buffer_cache_ports.cpp index 4971aff31c..3fb2dea4b1 100644 --- a/src/tests/video_core/buffer_cache_ports.cpp +++ b/src/tests/video_core/buffer_cache_ports.cpp @@ -26,8 +26,10 @@ #include "tests/core/system_test_context.h" #include "video_core/buffer_cache/buffer_cache.h" #include "video_core/buffer_cache/memory_tracker_base.h" +#include "video_core/buffer_cache/upload_order.h" #include "video_core/control/channel_state.h" #include "video_core/gpu.h" +#include "video_core/nx2_list_stats.h" #include "video_core/renderer_null/null_rasterizer.h" namespace { @@ -83,11 +85,9 @@ struct Runtime { return submitted_tick + 1; } void WaitSubmittedTick(u64) {} - void ClearBuffer(Buffer& buffer, u32 offset, size_t size, u32 value) { - std::memset(buffer.bytes.data() + offset, static_cast(value), size); - } - bool CanReorderUpload(const Buffer&, std::span) const { - return false; + VideoCommon::UploadOrder DecideUploadOrder(const Buffer&, + std::span) const { + return {}; } Staging DownloadStagingBuffer(u64 size, bool = false) { // Kept until the end: read downloads hold theirs past the next request. @@ -100,6 +100,22 @@ struct Runtime { return DownloadStagingBuffer(size); } void FreeDeferredStagingBuffer(Staging&) {} + /// Buffer to buffer, as when a new buffer takes over the ones it overlaps. + void CopyBuffer(Buffer& dst, Buffer& src, std::span copies, + bool) { + for (const auto& copy : copies) { + REQUIRE(copy.src_offset + copy.size <= src.bytes.size()); + REQUIRE(copy.dst_offset + copy.size <= dst.bytes.size()); + std::memcpy(dst.bytes.data() + copy.dst_offset, src.bytes.data() + copy.src_offset, + copy.size); + } + } + void ClearBuffer(Buffer& buffer, u32 offset, size_t size, u32 value) { + REQUIRE(offset + size <= buffer.bytes.size()); + for (size_t i = 0; i + sizeof(value) <= size; i += sizeof(value)) { + std::memcpy(buffer.bytes.data() + offset + i, &value, sizeof(value)); + } + } void CopyBuffer(Buffer* dst, Buffer& src, std::span copies, bool) { for (const auto& copy : copies) { @@ -112,12 +128,8 @@ struct Runtime { ++downloads; } } - void CopyBuffer(Buffer& dst, Buffer& src, std::span copies, - bool) { - CopyBuffer(dst, &src, copies, true, true); - } void CopyBuffer(Buffer& dst, Buffer* src, std::span copies, bool, - bool) { + VideoCommon::UploadOrder) { for (const auto& copy : copies) { REQUIRE(copy.src_offset + copy.size <= src->bytes.size()); REQUIRE(copy.dst_offset + copy.size <= dst.bytes.size()); @@ -332,6 +344,26 @@ struct BufferCacheTestAccess { static u32 CpuMarks(Cache& cache) { return cache.memory_tracker.cpu_marks; } + static void MarkWritten(Cache& cache, BufferId id, DAddr address, u32 size) { + cache.MarkWrittenBuffer(id, address, size); + } + /// A dispatch or copy wrote [address, address + size) of buffer `id` (Nx2NoteListWrite). + static void ListWrite(Cache& cache, BufferId id, DAddr address, u64 size) { + cache.Nx2NoteListWrite(id, address, size); + } + static void UseGpuMemory(Cache& cache, Tegra::MemoryManager& memory) { + cache.gpu_memory = &memory; + } + static size_t ReadDownloads(Cache& cache) { + return cache.read_downloads.size(); + } + /// TEST (EDEN_NX2_READ_CHECK): every read download compared with a synchronous flush. + static void CheckEveryRead(Cache& cache) { + cache.read_check.every = 1; + } + static const auto& ReadChecks(Cache& cache) { + return cache.read_check; + } static void KeepDraw(KeepCache& cache, BufferCacheChannelInfo& channel, const Nx2FastDraw::Delta* delta, BufferId id, DAddr address, u32 size) { @@ -352,19 +384,6 @@ struct BufferCacheTestAccess { static void GpuWrite(KeepCache& cache, BufferId id, DAddr address, u32 size) { cache.MarkWrittenBuffer(id, address, size); } - static void MarkWritten(Cache& cache, BufferId id, DAddr address, u32 size) { - cache.MarkWrittenBuffer(id, address, size); - } - static size_t ReadDownloads(Cache& cache) { - return cache.read_downloads.size(); - } - /// Every read download compared with a synchronous flush (EDEN_NX2_READ_CHECK=1). - static void CheckEveryRead(Cache& cache) { - cache.read_check.every = 1; - } - static const auto& ReadChecks(Cache& cache) { - return cache.read_check; - } }; } // namespace VideoCommon @@ -373,16 +392,6 @@ namespace { using Access = VideoCommon::BufferCacheTestAccess; -/// EDEN_NX2_READ_CHECK for a test, restored after it. -struct ReadCheckScope { - explicit ReadCheckScope(u64 every) - : previous{VideoCore::AsyncReadback::g_read_check_every.exchange(every)} {} - ~ReadCheckScope() { - VideoCore::AsyncReadback::g_read_check_every = previous; - } - u64 previous; -}; - template struct BasicFixture { ScopedSystemTestContext context; @@ -439,6 +448,59 @@ struct BasicFixture { } }; +/// Hands FinishRead to a buffer cache, as RasterizerVulkan does. +class ReadRasterizer final : public VideoCore::RasterizerInterface { +public: + explicit ReadRasterizer(Cache& cache_) : cache{cache_} {} + + void FinishRead(DAddr addr, u64 size) override { + cache.FinishReadDownloads(addr, size); + } + void Draw(bool, u32) override {} + void DrawTexture() override {} + void Clear(u32) override {} + void DispatchCompute() override {} + void ResetCounter(VideoCommon::QueryType) override {} + void Query(GPUVAddr, VideoCommon::QueryType, VideoCommon::QueryPropertiesFlags, u32, + u32) override {} + void BindGraphicsUniformBuffer(size_t, u32, GPUVAddr, u32) override {} + void DisableGraphicsUniformBuffer(size_t, u32) override {} + void SignalFence(std::function&&) override {} + void SyncOperation(std::function&&) override {} + void SignalSyncPoint(u32) override {} + void SignalReference() override {} + void ReleaseFences(bool) override {} + void FlushAll() override {} + void FlushRegion(DAddr, u64, VideoCommon::CacheType) override {} + bool MustFlushRegion(DAddr, u64, VideoCommon::CacheType) override { + return false; + } + VideoCore::RasterizerDownloadArea GetFlushArea(DAddr addr, u64 size) override { + return {addr, addr + size, true}; + } + void InvalidateRegion(DAddr, u64, VideoCommon::CacheType) override {} + void OnCacheInvalidation(DAddr, u64) override {} + bool OnCPUWrite(DAddr, u64) override { + return false; + } + void InvalidateGPUCache() override {} + void UnmapMemory(DAddr, u64) override {} + void ModifyGPUMemory(size_t, GPUVAddr, u64) override {} + void FlushAndInvalidateRegion(DAddr, u64, VideoCommon::CacheType) override {} + void WaitForIdle() override {} + void FragmentBarrier() override {} + void TiledCacheBarrier() override {} + void FlushCommands() override {} + void TickFrame() override {} + Tegra::Engines::AccelerateDMAInterface& AccessAccelerateDMA() override { + std::terminate(); + } + void AccelerateInlineToMemory(GPUVAddr, size_t, std::span) override {} + +private: + Cache& cache; +}; + using Fixture = BasicFixture; void SetEnv(const char* name, const char* value) { @@ -514,59 +576,6 @@ VideoCommon::Nx2FastDraw::Delta Written(DAddr begin, DAddr end) { return delta; } -/// Hands FinishRead to a buffer cache, as RasterizerVulkan does. -class ReadRasterizer final : public VideoCore::RasterizerInterface { -public: - explicit ReadRasterizer(Cache& cache_) : cache{cache_} {} - - void FinishRead(DAddr addr, u64 size) override { - cache.FinishReadDownloads(addr, size); - } - void Draw(bool, u32) override {} - void DrawTexture() override {} - void Clear(u32) override {} - void DispatchCompute() override {} - void ResetCounter(VideoCommon::QueryType) override {} - void Query(GPUVAddr, VideoCommon::QueryType, VideoCommon::QueryPropertiesFlags, u32, - u32) override {} - void BindGraphicsUniformBuffer(size_t, u32, GPUVAddr, u32) override {} - void DisableGraphicsUniformBuffer(size_t, u32) override {} - void SignalFence(std::function&&) override {} - void SyncOperation(std::function&&) override {} - void SignalSyncPoint(u32) override {} - void SignalReference() override {} - void ReleaseFences(bool) override {} - void FlushAll() override {} - void FlushRegion(DAddr, u64, VideoCommon::CacheType) override {} - bool MustFlushRegion(DAddr, u64, VideoCommon::CacheType) override { - return false; - } - VideoCore::RasterizerDownloadArea GetFlushArea(DAddr addr, u64 size) override { - return {addr, addr + size, true}; - } - void InvalidateRegion(DAddr, u64, VideoCommon::CacheType) override {} - void OnCacheInvalidation(DAddr, u64) override {} - bool OnCPUWrite(DAddr, u64) override { - return false; - } - void InvalidateGPUCache() override {} - void UnmapMemory(DAddr, u64) override {} - void ModifyGPUMemory(size_t, GPUVAddr, u64) override {} - void FlushAndInvalidateRegion(DAddr, u64, VideoCommon::CacheType) override {} - void WaitForIdle() override {} - void FragmentBarrier() override {} - void TiledCacheBarrier() override {} - void FlushCommands() override {} - void TickFrame() override {} - Tegra::Engines::AccelerateDMAInterface& AccessAccelerateDMA() override { - std::terminate(); - } - void AccelerateInlineToMemory(GPUVAddr, size_t, std::span) override {} - -private: - Cache& cache; -}; - } // namespace TEST_CASE("Storage binding cache tracks map changes and descriptor dependencies", @@ -813,6 +822,175 @@ TEST_CASE("Command page readers remain ordered during batch completion", "[buffe REQUIRE(f.runtime.downloads == 1); } +namespace { + +/// Makes `page` a command list page for one test (nx2.h keeps them for the session). +struct ListPage { + explicit ListPage(DAddr page) { + Tegra::Nx2ClearGpuWrittenPages(); + Tegra::Nx2MarkCommandListPage(page); + } + ~ListPage() { + Tegra::Nx2ClearGpuWrittenPages(); + } + ListPage(const ListPage&) = delete; + ListPage& operator=(const ListPage&) = delete; +}; + +bool AllBytes(std::span bytes, u8 value) { + return std::ranges::all_of(bytes, [value](u8 byte) { return byte == value; }); +} + +} // namespace + +TEST_CASE("Command lists on words no GPU work wrote read guest memory, not the page's copy", + "[buffer_ports]") { + Fixture f; + const ListPage list_page{0x20000}; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + std::ranges::fill(Access::Get(*f.cache, id).bytes, 0x77); + std::memset(f.bytes + 0x10000, 0x11, 0x1000); + // The page's first list drained the split and flushed the page (FlushGpuWrittenCommands). + f.cache->Nx2NoteListSynced(0x20000); + // A dispatch writes bytes 0x800-0x900 of the page; the copy taken after it is on its way. + f.cache->Nx2CollectListCopies(true); + Access::ListWrite(*f.cache, id, 0x20800, 0x100); + REQUIRE(f.cache->Nx2CopyListPages(f.runtime.CurrentTick())); + f.cache->Nx2CollectListCopies(false); + using Read = Cache::Nx2ListRead; + std::array list{}; + REQUIRE(f.cache->Nx2ReadListCopy(0x20100, list, true) == Read::Synced); + REQUIRE(f.cache->Nx2ReadListCopy(0x207f0, list, true) == Read::Synced); + // The GPU completes the copy. A list on the dispatch's words reads them from it; one that + // starts before them reads its other words from guest memory. + f.runtime.Finish(); + REQUIRE(f.cache->Nx2ReadListCopy(0x20800, list, true) == Read::Copied); + REQUIRE(AllBytes(list, 0x77)); + std::array spanning{}; + REQUIRE(f.cache->Nx2ReadListCopy(0x207f0, spanning, true) == Read::Copied); + REQUIRE(AllBytes(std::span{spanning}.first(16), 0x11)); + REQUIRE(AllBytes(std::span{spanning}.last(16), 0x77)); +} + +TEST_CASE("Command list pages the front has not flushed read every word from the copy", + "[buffer_ports]") { + Fixture f; + const ListPage list_page{0x20000}; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + std::ranges::fill(Access::Get(*f.cache, id).bytes, 0x77); + std::memset(f.bytes + 0x10000, 0x11, 0x1000); + // GPU work may have written the page before it became a command list page: only the first + // write since is noted, and guest memory need not hold the rest. + f.cache->Nx2CollectListCopies(true); + Access::ListWrite(*f.cache, id, 0x20800, 0x100); + REQUIRE(f.cache->Nx2CopyListPages(f.runtime.CurrentTick())); + f.cache->Nx2CollectListCopies(false); + f.runtime.Finish(); + std::array list{}; + REQUIRE(f.cache->Nx2ReadListCopy(0x20100, list, true) == Cache::Nx2ListRead::Copied); + REQUIRE(AllBytes(list, 0x77)); + REQUIRE(f.cache->Nx2ReadListCopy(0x207f0, list, true) == Cache::Nx2ListRead::Copied); + REQUIRE(AllBytes(list, 0x77)); +} + +TEST_CASE("Host writes give command list words back to guest memory and retire the page's copy", + "[buffer_ports]") { + Fixture f; + const ListPage list_page{0x20000}; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + f.cache->Nx2NoteListSynced(0x20000); + f.cache->Nx2CollectListCopies(true); + Access::ListWrite(*f.cache, id, 0x20800, 0x100); + REQUIRE(f.cache->Nx2CopyListPages(f.runtime.CurrentTick())); + f.cache->Nx2CollectListCopies(false); + f.runtime.Finish(); + using Read = Cache::Nx2ListRead; + std::array list{}; + REQUIRE(f.cache->Nx2ReadListCopy(0x20840, list, true) == Read::Copied); + // The CPU, an inline upload or a copy of bytes guest memory holds writes 0x800-0x840 whole, + // and two bytes of each of the words at 0x880 and 0x884. + f.cache->Nx2NoteListHostWrite(0x20800, 0x40); + f.cache->Nx2NoteListHostWrite(0x20882, 4); + REQUIRE(f.cache->Nx2ReadListCopy(0x20800, list, true) == Read::Synced); + REQUIRE(f.cache->Nx2ReadListCopy(0x20830, list, true) == Read::Synced); + // GPU-written words it did not write whole: the copy no longer holds the page. + REQUIRE(f.cache->Nx2ReadListCopy(0x20840, list, true) == Read::Unknown); + REQUIRE(f.cache->Nx2ReadListCopy(0x20880, list, true) == Read::Unknown); +} + +TEST_CASE("A flushed command list page reads guest memory until GPU work writes it again", + "[buffer_ports]") { + Fixture f; + const ListPage list_page{0x20000}; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + f.cache->Nx2CollectListCopies(true); + Access::ListWrite(*f.cache, id, 0x20000, 0x1000); + REQUIRE(f.cache->Nx2CopyListPages(f.runtime.CurrentTick())); + using Read = Cache::Nx2ListRead; + std::array list{}; + REQUIRE(f.cache->Nx2ReadListCopy(0x20800, list, true) == Read::Copied); + // The front drained the split and flushed the page (DmaPusher::FlushGpuWrittenCommands). + f.cache->Nx2NoteListSynced(0x20000); + REQUIRE(f.cache->Nx2ReadListCopy(0x20800, list, true) == Read::Synced); + Access::ListWrite(*f.cache, id, 0x20900, 0x40); + REQUIRE(f.cache->Nx2CopyListPages(f.runtime.CurrentTick())); + f.cache->Nx2CollectListCopies(false); + f.runtime.Finish(); + REQUIRE(f.cache->Nx2ReadListCopy(0x20800, list, true) == Read::Synced); + REQUIRE(f.cache->Nx2ReadListCopy(0x20900, list, true) == Read::Copied); +} + +TEST_CASE("Copy engine clears give command list words back to guest memory", "[buffer_ports]") { + Fixture f; + const ListPage list_page{0x20000}; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + Tegra::MemoryManager gpu_memory{f.system, f.device, 32, 1ULL << 31, 16}; + gpu_memory.BindRasterizer(&f.rasterizer); + gpu_memory.Map(0x200000, 0x20000, 0x10000, Tegra::PTEKind::PITCH, true); + Access::UseGpuMemory(*f.cache, gpu_memory); + f.cache->Nx2NoteListSynced(0x20000); + f.cache->Nx2CollectListCopies(true); + Access::ListWrite(*f.cache, id, 0x20800, 0x100); + REQUIRE(f.cache->Nx2CopyListPages(f.runtime.CurrentTick())); + f.cache->Nx2CollectListCopies(false); + f.runtime.Finish(); + // 16 words cleared (MaxwellDMA's remapped constant fill writes them to guest memory too). + REQUIRE(f.cache->DMAClear(0x200800, 16, 0)); + REQUIRE(AllBytes(std::span{Access::Get(*f.cache, id).bytes}.subspan(0x800, 0x40), 0)); + using Read = Cache::Nx2ListRead; + std::array list{}; + REQUIRE(f.cache->Nx2ReadListCopy(0x20830, list, true) == Read::Synced); + REQUIRE(f.cache->Nx2ReadListCopy(0x20840, list, true) == Read::Unknown); +} + +TEST_CASE("EDEN_NX2_LIST_VERIFY reads command lists as before, from the page's whole copy", + "[buffer_ports]") { + Fixture f; + const ListPage list_page{0x20000}; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + std::ranges::fill(Access::Get(*f.cache, id).bytes, 0x77); + std::memset(f.bytes + 0x10000, 0x11, 0x1000); + f.cache->Nx2NoteListSynced(0x20000); + f.cache->Nx2CollectListCopies(true); + Access::ListWrite(*f.cache, id, 0x20800, 0x100); + REQUIRE(f.cache->Nx2CopyListPages(f.runtime.CurrentTick())); + f.cache->Nx2CollectListCopies(false); + f.runtime.Finish(); + using Read = Cache::Nx2ListRead; + std::array list{}; + { + // Whatever words GPU work wrote: once it wrote the page, every list there reads the copy. + const Tegra::Nx2ListStats::ReferenceScope reference; + REQUIRE(f.cache->Nx2ReadListCopy(0x20100, list, true) == Read::Copied); + REQUIRE(AllBytes(list, 0x77)); + REQUIRE(f.cache->Nx2ReadListCopy(0x207f0, list, true) == Read::Copied); + REQUIRE(AllBytes(list, 0x77)); + } + REQUIRE(f.cache->Nx2ReadListCopy(0x20100, list, true) == Read::Synced); + REQUIRE(f.cache->Nx2ReadListCopy(0x207f0, list, true) == Read::Copied); + REQUIRE(AllBytes(std::span{list}.first(16), 0x11)); +} + TEST_CASE("Native uniform banks upload directly without overwriting GPU-owned bytes", "[buffer_ports]") { Fixture f; @@ -897,124 +1075,40 @@ TEST_CASE("Downloads keep CPU writes collected and not yet applied", "[buffer_po REQUIRE(f.bytes[0x21008] == 0x34); } -TEST_CASE("Uniform keep binds unwritten uniform buffers of delta draws without reading them", +TEST_CASE("Read downloads stay GPU-modified until the reading thread writes them", "[buffer_ports]") { - KeepFixture f{nullptr}; - using Delta = VideoCommon::Nx2FastDraw::Delta; - const Delta unchanged{}; - std::memset(f.UboBytes(), 1, KeepFixture::UboSize); - f.Draw(nullptr); - f.Draw(&unchanged); - REQUIRE(f.Counts().kept == 1); - REQUIRE(f.runtime.allocations.size() == 1); - - // Inline constant data over the range, then a rebind of the slot. - f.UboBytes()[0x10] = 2; - const Delta written = Written(KeepFixture::Ubo + 0x10, KeepFixture::Ubo + 0x14); - f.Draw(&written); - REQUIRE(f.Counts().kept == 1); - REQUIRE(f.runtime.allocations.size() == 2); - Delta rebound{}; - rebound.binds[0] = 1; - f.Draw(&rebound); - REQUIRE(f.Counts().kept == 1); - f.Draw(&unchanged); - REQUIRE(f.Counts().kept == 2); - - // Data written next to the range, and GPU writes elsewhere, leave it kept. - const Delta beside = Written(KeepFixture::Ubo + 0x100, KeepFixture::Ubo + 0x140); - f.Draw(&beside); - REQUIRE(f.Counts().kept == 3); - Access::GpuWrite(*f.cache, f.id, 0x18000, 0x100); - f.Draw(&unchanged); - REQUIRE(f.Counts().kept == 4); - - // A GPU write over the range: the original path binds it from the buffer from now on. - Access::GpuWrite(*f.cache, f.id, KeepFixture::Ubo + 0x80, 4); - const size_t allocations = f.runtime.allocations.size(); - Access::KeepDraw(*f.cache, *f.channel, &unchanged, f.id, KeepFixture::Ubo, KeepFixture::UboSize); - Access::KeepDraw(*f.cache, *f.channel, &unchanged, f.id, KeepFixture::Ubo, KeepFixture::UboSize); - REQUIRE(f.Counts().kept == 4); - REQUIRE(f.runtime.allocations.size() == allocations); -} - -TEST_CASE("Uniform keep ends at CPU writes the caches are told about", "[buffer_ports]") { - KeepFixture f{nullptr}; - const VideoCommon::Nx2FastDraw::Delta unchanged{}; - f.Draw(nullptr); - f.Draw(&unchanged); - REQUIRE(f.Counts().kept == 1); - f.UboBytes()[0x20] = 7; - f.cache->WriteMemory(KeepFixture::Ubo + 0x20, 1); - f.Draw(&unchanged); - REQUIRE(f.Counts().kept == 1); - f.cache->WriteMemory(0x18000, 0x10); - f.Draw(&unchanged); - REQUIRE(f.Counts().kept == 2); -} - -TEST_CASE("Uniform keep never keeps across command buffers", "[buffer_ports]") { - KeepFixture f{nullptr}; - const VideoCommon::Nx2FastDraw::Delta unchanged{}; - f.Draw(nullptr); - ++f.runtime.tick; - f.Draw(&unchanged); - REQUIRE(f.Counts().kept == 0); - REQUIRE(f.runtime.allocations.size() == 2); - f.Draw(nullptr); - f.Draw(&unchanged); - REQUIRE(f.Counts().kept == 1); -} - -TEST_CASE("Uniform keep verify counts guest writes the delta did not report", - "[buffer_ports]") { - KeepFixture f{"EDEN_NX2_UBO_KEEP_VERIFY"}; - const VideoCommon::Nx2FastDraw::Delta unchanged{}; - f.Draw(nullptr); - f.Draw(&unchanged); - REQUIRE(f.Counts().verified == 1); - REQUIRE(f.Counts().mismatches == 0); - f.UboBytes()[3] = 9; - f.Draw(&unchanged); - REQUIRE(f.Counts().verified == 2); - REQUIRE(f.Counts().mismatches == 1); - REQUIRE(f.Counts().kept == 0); -} - -TEST_CASE("Read downloads stay GPU-modified until the reading thread writes them", - "[buffer_ports]") { - Fixture f; - const bool mapped = GENERATE(false, true); - const auto id = Access::Add(*f.cache, 0x20000, 0x10000, mapped); - std::ranges::fill(Access::Get(*f.cache, id).bytes, 0x12); - Access::Dirty(*f.cache, 0x20000, 0x2000); - f.cache->DownloadMemoryForRead(0x20000, 0x1000); - REQUIRE(f.runtime.finishes == 0); - REQUIRE(f.runtime.downloads == 1); - REQUIRE(f.runtime.host_read_barriers == 1); - REQUIRE(VideoCore::g_read_downloads == 1); - REQUIRE(f.bytes[0x10000] == 0); - REQUIRE(Access::Pending(*f.cache, 0x20000, 0x1000)); - // Another read of the range waits for the same copy, and its flush copies nothing. - const auto area = f.cache->GetFlushArea(0x20800, 4); - REQUIRE(area); - REQUIRE_FALSE(area->preemtive); - REQUIRE(area->in_flight); - f.cache->DownloadMemoryForRead(0x20000, 0x1000); - REQUIRE(f.runtime.downloads == 1); - REQUIRE(f.runtime.submitted_tick == 1); - f.cache->FinishReadDownloads(0x20800, 4); - REQUIRE(f.runtime.host_read_waits == 1); - REQUIRE(VideoCore::g_read_downloads == 0); - REQUIRE(f.bytes[0x10000] == 0x12); - REQUIRE(f.bytes[0x10fff] == 0x12); - REQUIRE(f.bytes[0x11000] == 0); - REQUIRE_FALSE(Access::Pending(*f.cache, 0x20000, 0x1000)); - REQUIRE(Access::Pending(*f.cache, 0x21000, 0x1000)); - REQUIRE(f.cache->GetFlushArea(0x20800, 4)->preemtive); - f.cache->FinishReadDownloads(0x20000, 0x2000); - REQUIRE(f.runtime.host_read_waits == 1); -} + Fixture f; + const bool mapped = GENERATE(false, true); + const auto id = Access::Add(*f.cache, 0x20000, 0x10000, mapped); + std::ranges::fill(Access::Get(*f.cache, id).bytes, 0x12); + Access::Dirty(*f.cache, 0x20000, 0x2000); + f.cache->DownloadMemoryForRead(0x20000, 0x1000); + REQUIRE(f.runtime.finishes == 0); + REQUIRE(f.runtime.downloads == 1); + REQUIRE(f.runtime.host_read_barriers == 1); + REQUIRE(VideoCore::g_read_downloads == 1); + REQUIRE(f.bytes[0x10000] == 0); + REQUIRE(Access::Pending(*f.cache, 0x20000, 0x1000)); + // Another read of the range waits for the same copy, and its flush copies nothing. + const auto area = f.cache->GetFlushArea(0x20800, 4); + REQUIRE(area); + REQUIRE_FALSE(area->preemtive); + REQUIRE(area->in_flight); + f.cache->DownloadMemoryForRead(0x20000, 0x1000); + REQUIRE(f.runtime.downloads == 1); + REQUIRE(f.runtime.submitted_tick == 1); + f.cache->FinishReadDownloads(0x20800, 4); + REQUIRE(f.runtime.host_read_waits == 1); + REQUIRE(VideoCore::g_read_downloads == 0); + REQUIRE(f.bytes[0x10000] == 0x12); + REQUIRE(f.bytes[0x10fff] == 0x12); + REQUIRE(f.bytes[0x11000] == 0); + REQUIRE_FALSE(Access::Pending(*f.cache, 0x20000, 0x1000)); + REQUIRE(Access::Pending(*f.cache, 0x21000, 0x1000)); + REQUIRE(f.cache->GetFlushArea(0x20800, 4)->preemtive); + f.cache->FinishReadDownloads(0x20000, 0x2000); + REQUIRE(f.runtime.host_read_waits == 1); +} TEST_CASE("Read downloads keep CPU writes made while they were on their way", "[buffer_ports]") { Fixture f; @@ -1039,36 +1133,6 @@ TEST_CASE("Read downloads keep CPU writes made while they were on their way", "[ REQUIRE_FALSE(Access::Pending(*f.cache, 0x20000, 0x1000)); } -TEST_CASE("Newer bytes that guest memory doesn't hold yet land the read downloads under them first", - "[buffer_ports]") { - Fixture f; - const auto id = Access::Add(*f.cache, 0x20000, 0x10000); - std::ranges::fill(Access::Get(*f.cache, id).bytes, 0x12); - Access::Dirty(*f.cache, 0x20000, 0x1000); - f.cache->DownloadMemoryForRead(0x20000, 0x1000); - auto& counters = VideoCore::AsyncReadback::g_counters; - const u64 landings = counters.clear_landings.load(); - const u64 waits = counters.clear_waits.load(); - SECTION("an indirect dispatch's arguments, written by the GPU and discarded") { - (void)f.cache->ObtainCPUBuffer(0x20040, 12, - VideoCommon::ObtainBufferSynchronize::FullSynchronize, - VideoCommon::ObtainBufferOperation::DiscardWrite); - } - SECTION("a DMA copy's destination (ClearDownload)") { - f.cache->ClearDownload(0x20040, 12); - } - // The read that took the download finds the GPU's bytes, as its synchronous flush wrote them. - f.cache->FinishReadDownloads(0x20000, 0x1000); - REQUIRE(f.bytes[0x10040] == 0x12); - REQUIRE(f.bytes[0x1004b] == 0x12); - REQUIRE(f.bytes[0x10fff] == 0x12); - REQUIRE(f.runtime.host_read_waits == 1); - REQUIRE(VideoCore::g_read_downloads == 0); - // The landing went on the GPU thread and waited for the host GPU (the profile line counts it). - REQUIRE(counters.clear_landings.load() - landings == 1); - REQUIRE(counters.clear_waits.load() - waits == 1); -} - TEST_CASE("Read downloads land oldest first and leave newer GPU writes to download", "[buffer_ports]") { Fixture f; @@ -1343,6 +1407,123 @@ VideoCore::RasterizerDownloadArea ReadWord(Fixture& f, DAddr address) { } } // namespace +TEST_CASE("Newer bytes not yet in guest memory land overlapping read downloads first", + "[buffer_ports]") { + Fixture f; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + std::ranges::fill(Access::Get(*f.cache, id).bytes, 0x12); + Access::Dirty(*f.cache, 0x20000, 0x1000); + f.cache->DownloadMemoryForRead(0x20000, 0x1000); + SECTION("indirect dispatch arguments discarded before the GPU writes them") { + (void)f.cache->ObtainCPUBuffer(0x20040, 12, + VideoCommon::ObtainBufferSynchronize::FullSynchronize, + VideoCommon::ObtainBufferOperation::DiscardWrite); + } + SECTION("a DMA copy destination") { + f.cache->ClearDownload(0x20040, 12); + } + f.cache->FinishReadDownloads(0x20000, 0x1000); + REQUIRE(f.bytes[0x10040] == 0x12); + REQUIRE(f.bytes[0x1004b] == 0x12); + REQUIRE(f.bytes[0x10fff] == 0x12); + REQUIRE(f.runtime.host_read_waits == 1); + REQUIRE(VideoCore::g_read_downloads == 0); +} + +namespace { +void ReportOnReadBackPage(Fixture& f, VideoCommon::BufferId id) { + for (u8 frame = 1; frame <= 2; ++frame) { + WriteByDispatch(f, id, 0x20000, 0x100, frame); + ReadWord(f, 0x20010); + } + REQUIRE(f.cache->NoteReportPending(0x20040, 4)); +} + +void WriteReportResult(Fixture& f, u32 result, bool noted) { + f.cache->FinishReadDownloads(0x20040, 4); + std::memcpy(f.bytes + 0x10040, &result, sizeof(result)); + if (noted) { + f.cache->FinishReport(0x20040, 4); + } +} +} // namespace + +TEST_CASE("An older query result leaves overlapping GPU writes to be read again", + "[buffer_ports]") { + Fixture f; + const Nx2TitleScope nx2_title; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + ReportOnReadBackPage(f, id); + WriteByDispatch(f, id, 0x20000, 0x100, 3); + REQUIRE(Access::ReadDownloads(*f.cache) == 1); + SECTION("its early download lands at the result's own wait") {} + SECTION("its early download landed before the result") { + f.runtime.Complete(f.runtime.submitted_tick); + f.cache->NoteWorkRecorded(true); + REQUIRE(Access::ReadDownloads(*f.cache) == 0); + REQUIRE(f.bytes[0x10040] == 3); + } + WriteReportResult(f, 0xcafef00d, true); + REQUIRE_FALSE(ReadWord(f, 0x20040).preemtive); + REQUIRE(f.bytes[0x10040] == 3); + REQUIRE(f.bytes[0x10043] == 3); +} + +TEST_CASE("A synced payload does not end an older deferred report's tracking", + "[buffer_ports]") { + Fixture f; + const Nx2TitleScope nx2_title; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + ReportOnReadBackPage(f, id); + WriteByDispatch(f, id, 0x20000, 0x100, 3); + // #874's synced payload runs now, ahead of the older report. CounterReport does not + // register or finish tracking for this callback, just as for an immediate payload. + WriteReportResult(f, 0x11111111, false); + REQUIRE_FALSE(ReadWord(f, 0x20040).preemtive); + REQUIRE(f.bytes[0x10040] == 3); + WriteReportResult(f, 0x22222222, true); + REQUIRE_FALSE(ReadWord(f, 0x20040).preemtive); + REQUIRE(f.bytes[0x10040] == 3); +} + +TEST_CASE("A query result newer than the GPU writes under it stays in guest memory", + "[buffer_ports]") { + Fixture f; + const Nx2TitleScope nx2_title; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + for (u8 frame = 1; frame <= 2; ++frame) { + WriteByDispatch(f, id, 0x20000, 0x100, frame); + ReadWord(f, 0x20010); + } + WriteByDispatch(f, id, 0x20000, 0x100, 3); + const bool noted = f.cache->NoteReportPending(0x20040, 4); + REQUIRE(noted); + WriteReportResult(f, 0xcafef00d, noted); + REQUIRE(ReadWord(f, 0x20040).preemtive); + u32 value{}; + std::memcpy(&value, f.bytes + 0x10040, sizeof(value)); + REQUIRE(value == 0xcafef00d); +} + +TEST_CASE("An untracked result ends no other result's tracking", "[buffer_ports]") { + Fixture f; + const Nx2TitleScope nx2_title; + const auto id = Access::Add(*f.cache, 0x20000, 0x10000); + const bool r1 = f.cache->NoteReportPending(0x20040, 4); + REQUIRE_FALSE(r1); + for (u8 frame = 1; frame <= 2; ++frame) { + WriteByDispatch(f, id, 0x20000, 0x100, frame); + ReadWord(f, 0x20010); + } + const bool r2 = f.cache->NoteReportPending(0x20040, 4); + REQUIRE(r2); + WriteReportResult(f, 0x11111111, r1); + WriteByDispatch(f, id, 0x20000, 0x100, 3); + WriteReportResult(f, 0x22222222, r2); + REQUIRE_FALSE(ReadWord(f, 0x20040).preemtive); + REQUIRE(f.bytes[0x10040] == 3); +} + TEST_CASE("Pages read after each GPU write get their downloads right behind the writes", "[buffer_ports]") { Fixture f; @@ -1479,94 +1660,6 @@ TEST_CASE("GPU writes to guest memory land the early downloads under them first" REQUIRE(ReadWord(f, 0x20040).preemtive); } -namespace { -/// A page read after each dispatch's write, which gets its next write's early download, and a -/// query result to 0x20040 that the query cache writes at the report's fence. -void ReportOnReadBackPage(Fixture& f, VideoCommon::BufferId id) { - for (u8 frame = 1; frame <= 2; ++frame) { - WriteByDispatch(f, id, 0x20000, 0x100, frame); - ReadWord(f, 0x20010); - } - REQUIRE(f.cache->NoteReportPending(0x20040, 4)); -} - -/// The report's fence callback (QueryCacheBase::CounterReport) writes its result, and ends it if -/// the buffer cache kept it (`noted`). -void WriteReportResult(Fixture& f, u32 result, bool noted) { - f.cache->FinishReadDownloads(0x20040, 4); - std::memcpy(f.bytes + 0x10040, &result, sizeof(result)); - if (noted) { - f.cache->FinishReport(0x20040, 4); - } -} -} // namespace - -TEST_CASE("A query result older than a GPU write it lands over leaves the write to be read again", - "[buffer_ports]") { - Fixture f; - const Nx2TitleScope nx2_title; - const auto id = Access::Add(*f.cache, 0x20000, 0x10000); - ReportOnReadBackPage(f, id); - // A dispatch writes the result's words after the report, before its fence. - WriteByDispatch(f, id, 0x20000, 0x100, 3); - REQUIRE(Access::ReadDownloads(*f.cache) == 1); - SECTION("its early download landed by the result's own wait") {} - SECTION("its early download landed before, once the host GPU ran it") { - f.runtime.Complete(f.runtime.submitted_tick); - f.cache->NoteWorkRecorded(true); - REQUIRE(Access::ReadDownloads(*f.cache) == 0); - REQUIRE(f.bytes[0x10040] == 3); - } - WriteReportResult(f, 0xcafef00d, true); - // The dispatch's words are newer: the next read downloads them again. - REQUIRE_FALSE(ReadWord(f, 0x20040).preemtive); - REQUIRE(f.bytes[0x10040] == 3); - REQUIRE(f.bytes[0x10043] == 3); -} - -TEST_CASE("A query result newer than the GPU writes under it stays in guest memory", - "[buffer_ports]") { - Fixture f; - const Nx2TitleScope nx2_title; - const auto id = Access::Add(*f.cache, 0x20000, 0x10000); - // The dispatch writes the words before the report. - for (u8 frame = 1; frame <= 2; ++frame) { - WriteByDispatch(f, id, 0x20000, 0x100, frame); - ReadWord(f, 0x20010); - } - WriteByDispatch(f, id, 0x20000, 0x100, 3); - const bool noted = f.cache->NoteReportPending(0x20040, 4); - REQUIRE(noted); - WriteReportResult(f, 0xcafef00d, noted); - REQUIRE(ReadWord(f, 0x20040).preemtive); - u32 value{}; - std::memcpy(&value, f.bytes + 0x10040, sizeof(value)); - REQUIRE(value == 0xcafef00d); -} - -TEST_CASE("A result the buffer cache didn't keep ends no other result's tracking", - "[buffer_ports]") { - Fixture f; - const Nx2TitleScope nx2_title; - const auto id = Access::Add(*f.cache, 0x20000, 0x10000); - // R1 at 0x20040, reported before the page gets early downloads: not kept. - const bool r1 = f.cache->NoteReportPending(0x20040, 4); - REQUIRE_FALSE(r1); - for (u8 frame = 1; frame <= 2; ++frame) { - WriteByDispatch(f, id, 0x20000, 0x100, frame); - ReadWord(f, 0x20010); - } - // R2 at the same words, then R1's callback. - const bool r2 = f.cache->NoteReportPending(0x20040, 4); - REQUIRE(r2); - WriteReportResult(f, 0x11111111, r1); - // A dispatch writes the words after R2's report, then R2's callback. - WriteByDispatch(f, id, 0x20000, 0x100, 3); - WriteReportResult(f, 0x22222222, r2); - REQUIRE_FALSE(ReadWord(f, 0x20040).preemtive); - REQUIRE(f.bytes[0x10040] == 3); -} - TEST_CASE("Finished early downloads land a few draws later, or at the next dispatch", "[buffer_ports]") { Fixture f; @@ -2123,118 +2216,95 @@ TEST_CASE("Read download check leaves out early downloaded bytes an upload repla REQUIRE(Access::ReadChecks(*f.cache).uncovered == 0); } -TEST_CASE("EDEN_NX2_READ_CHECK values select how many read downloads are checked", +TEST_CASE("Uniform keep binds unwritten uniform buffers of delta draws without reading them", "[buffer_ports]") { - using VideoCore::AsyncReadback::ParseReadCheck; - REQUIRE(ParseReadCheck(nullptr) == 0); - REQUIRE(ParseReadCheck("") == 0); - REQUIRE(ParseReadCheck("0") == 0); - REQUIRE(ParseReadCheck("1") == 1); - REQUIRE(ParseReadCheck("16") == 16); -} + KeepFixture f{nullptr}; + using Delta = VideoCommon::Nx2FastDraw::Delta; + const Delta unchanged{}; + std::memset(f.UboBytes(), 1, KeepFixture::UboSize); + f.Draw(nullptr); + f.Draw(&unchanged); + REQUIRE(f.Counts().kept == 1); + REQUIRE(f.runtime.allocations.size() == 1); -TEST_CASE("EDEN_NX2_READ_CHECK=1 compares every read download", "[buffer_ports]") { - Fixture f; - const ReadCheckScope check{1}; - const auto id = Access::Add(*f.cache, 0x20000, 0x10000); - std::ranges::fill(Access::Get(*f.cache, id).bytes, 0x12); - for (u32 read = 1; read <= 3; ++read) { - Access::MarkWritten(*f.cache, id, 0x20000, 0x100); - f.cache->DownloadMemoryForRead(0x20000, 0x1000); - f.cache->FinishReadDownloads(0x20000, 0x1000); - REQUIRE(Access::ReadChecks(*f.cache).checked == read); - } - REQUIRE(Access::ReadChecks(*f.cache).compared == 0x300); - REQUIRE(Access::ReadChecks(*f.cache).mismatched == 0); - REQUIRE(Access::ReadChecks(*f.cache).uncovered == 0); -} + // Inline constant data over the range, then a rebind of the slot. + f.UboBytes()[0x10] = 2; + const Delta written = Written(KeepFixture::Ubo + 0x10, KeepFixture::Ubo + 0x14); + f.Draw(&written); + REQUIRE(f.Counts().kept == 1); + REQUIRE(f.runtime.allocations.size() == 2); + Delta rebound{}; + rebound.binds[0] = 1; + f.Draw(&rebound); + REQUIRE(f.Counts().kept == 1); + f.Draw(&unchanged); + REQUIRE(f.Counts().kept == 2); -namespace { -/// Two read downloads, and a GPU write nothing records into the first one's bytes: the second -/// download's check finds 0x10 bytes outdated once both land. -VideoCommon::BufferId OutdatedReadDownloads(Fixture& f) { - const auto id = Access::Add(*f.cache, 0x20000, 0x10000); - auto& buffer = Access::Get(*f.cache, id); - std::ranges::fill(buffer.bytes, 0x12); - Access::MarkWritten(*f.cache, id, 0x20000, 0x100); - f.cache->DownloadMemoryForRead(0x20000, 0x1000); - f.runtime.pending.push_back( - [&buffer] { std::fill_n(buffer.bytes.begin() + 0x40, 0x10, u8{0x56}); }); - Access::MarkWritten(*f.cache, id, 0x20800, 0x10); - f.cache->DownloadMemoryForRead(0x20000, 0x1000); - return id; + // Data written next to the range, and GPU writes elsewhere, leave it kept. + const Delta beside = Written(KeepFixture::Ubo + 0x100, KeepFixture::Ubo + 0x140); + f.Draw(&beside); + REQUIRE(f.Counts().kept == 3); + Access::GpuWrite(*f.cache, f.id, 0x18000, 0x100); + f.Draw(&unchanged); + REQUIRE(f.Counts().kept == 4); + + // A GPU write over the range: the original path binds it from the buffer from now on. + Access::GpuWrite(*f.cache, f.id, KeepFixture::Ubo + 0x80, 4); + const size_t allocations = f.runtime.allocations.size(); + Access::KeepDraw(*f.cache, *f.channel, &unchanged, f.id, KeepFixture::Ubo, KeepFixture::UboSize); + Access::KeepDraw(*f.cache, *f.channel, &unchanged, f.id, KeepFixture::Ubo, KeepFixture::UboSize); + REQUIRE(f.Counts().kept == 4); + REQUIRE(f.runtime.allocations.size() == allocations); } -/// The guest thread whose read waits for its downloads (GPU::OnCPURead). -struct ReadingThreadScope { - ReadingThreadScope() { - VideoCore::AsyncReadback::t_reading = true; - } - ~ReadingThreadScope() { - VideoCore::AsyncReadback::t_reading = false; - } -}; +TEST_CASE("Uniform keep ends at CPU writes the caches are told about", "[buffer_ports]") { + KeepFixture f{nullptr}; + const VideoCommon::Nx2FastDraw::Delta unchanged{}; + f.Draw(nullptr); + f.Draw(&unchanged); + REQUIRE(f.Counts().kept == 1); + f.UboBytes()[0x20] = 7; + f.cache->WriteMemory(KeepFixture::Ubo + 0x20, 1); + f.Draw(&unchanged); + REQUIRE(f.Counts().kept == 1); + f.cache->WriteMemory(0x18000, 0x10); + f.Draw(&unchanged); + REQUIRE(f.Counts().kept == 2); -u64 MismatchedBy(Fixture& f, VideoCore::AsyncReadback::Landing landing) { - return Access::ReadChecks(*f.cache).mismatched_by_landing[static_cast(landing)]; + // Losing the write list is conservative even when the recorded ranges are elsewhere. + for (u32 write = 0; write < 9; ++write) { + f.cache->WriteMemory(0x18000 + write * 0x10, 0x10); + } + f.Draw(&unchanged); + REQUIRE(f.Counts().kept == 2); + f.Draw(&unchanged); + REQUIRE(f.Counts().kept == 3); } -} // namespace -TEST_CASE("The read check names who wrote a differing download to guest memory", - "[buffer_ports]") { - using VideoCore::AsyncReadback::Landing; - Fixture f; - const ReadCheckScope check{1}; - OutdatedReadDownloads(f); - SECTION("the reading thread") { - const ReadingThreadScope reading; - f.cache->FinishReadDownloads(0x20000, 0x1000); - REQUIRE(MismatchedBy(f, Landing::Reader) == 0x10); - } - SECTION("a write to guest memory from the GPU's side") { - f.cache->FinishReadDownloads(0x20800, 4); - REQUIRE(MismatchedBy(f, Landing::GpuSideWrite) == 0x10); - } - SECTION("the next read download, once the host GPU ran them") { - f.runtime.Complete(f.runtime.submitted_tick); - f.cache->DownloadMemoryForRead(0x30000, 0x1000); - REQUIRE(MismatchedBy(f, Landing::Completed) == 0x10); - } - REQUIRE(Access::ReadDownloads(*f.cache) == 0); - REQUIRE(Access::ReadChecks(*f.cache).checked == 2); - REQUIRE(Access::ReadChecks(*f.cache).mismatched == 0x10); +TEST_CASE("Uniform keep never keeps across command buffers", "[buffer_ports]") { + KeepFixture f{nullptr}; + const VideoCommon::Nx2FastDraw::Delta unchanged{}; + f.Draw(nullptr); + ++f.runtime.tick; + f.Draw(&unchanged); + REQUIRE(f.Counts().kept == 0); + REQUIRE(f.runtime.allocations.size() == 2); + f.Draw(nullptr); + f.Draw(&unchanged); + REQUIRE(f.Counts().kept == 1); } -TEST_CASE("Async read-back counts the reading thread's and the GPU side's waits apart", +TEST_CASE("Uniform keep verify counts guest writes the delta did not report", "[buffer_ports]") { - auto& counters = VideoCore::AsyncReadback::g_counters; - Fixture f; - const auto id = Access::Add(*f.cache, 0x20000, 0x10000); - const u64 reader_waits = counters.reader_waits.load(); - const u64 gpu_side_waits = counters.gpu_side_waits.load(); - // The reading thread waits for a download the host GPU hasn't run. - Access::MarkWritten(*f.cache, id, 0x20000, 0x100); - f.cache->DownloadMemoryForRead(0x20000, 0x1000); - { - const ReadingThreadScope reading; - f.cache->FinishReadDownloads(0x20000, 0x1000); - } - REQUIRE(counters.reader_waits.load() - reader_waits == 1); - REQUIRE(counters.gpu_side_waits.load() == gpu_side_waits); - // A semaphore release over one falls back to a host GPU wait on the GPU's side. - Access::MarkWritten(*f.cache, id, 0x20000, 0x100); - f.cache->DownloadMemoryForRead(0x20000, 0x1000); - f.cache->FinishReadDownloads(0x20040, 4); - REQUIRE(counters.gpu_side_waits.load() - gpu_side_waits == 1); - // Run by the host GPU before the read: nothing waits. - Access::MarkWritten(*f.cache, id, 0x20000, 0x100); - f.cache->DownloadMemoryForRead(0x20000, 0x1000); - f.runtime.Complete(f.runtime.submitted_tick); - { - const ReadingThreadScope reading; - f.cache->FinishReadDownloads(0x20000, 0x1000); - } - REQUIRE(counters.reader_waits.load() - reader_waits == 1); - REQUIRE(counters.gpu_side_waits.load() - gpu_side_waits == 1); - REQUIRE(f.runtime.host_read_waits == 2); + KeepFixture f{"EDEN_NX2_UBO_KEEP_VERIFY"}; + const VideoCommon::Nx2FastDraw::Delta unchanged{}; + f.Draw(nullptr); + f.Draw(&unchanged); + REQUIRE(f.Counts().verified == 1); + REQUIRE(f.Counts().mismatches == 0); + f.UboBytes()[3] = 9; + f.Draw(&unchanged); + REQUIRE(f.Counts().verified == 2); + REQUIRE(f.Counts().mismatches == 1); + REQUIRE(f.Counts().kept == 0); } diff --git a/src/tests/video_core/fast_draw.cpp b/src/tests/video_core/fast_draw.cpp index bf5a0a6206..e4fbccbf61 100644 --- a/src/tests/video_core/fast_draw.cpp +++ b/src/tests/video_core/fast_draw.cpp @@ -345,6 +345,56 @@ TEST_CASE("Fast draw keeps base vertex and instance as draw parameters", "[fast_ CHECK(f.Reusable()); } +TEST_CASE("Fast draw keeps counter resets and the render enable condition light", "[fast_draw]") { + FastDrawFixture f; + f.DrawArrays(0, 3); + constexpr u32 clear_report = MAXWELL3D_REG_INDEX(clear_report_value); + f.Write(clear_report, f.Reg(clear_report) ^ 1); + f.DrawArrays(3, 3); + CHECK(f.Reusable()); + f.Write(MAXWELL3D_REG_INDEX(render_enable.mode), 1); // True: the next draws run + f.DrawArrays(6, 3); + CHECK(f.rasterizer.draws.size() == 3); + CHECK(f.Reusable()); +} + +TEST_CASE("Fast draw records a report's memory write instead of breaking on it", "[fast_draw]") { + DeltaFixture f; + f.Write(MAXWELL3D_REG_INDEX(tex_header.address_high), 0); + f.Write(MAXWELL3D_REG_INDEX(tex_header.address_low), 0x40000); + f.Write(MAXWELL3D_REG_INDEX(tex_header.limit), 7); + const auto report = [&](u32 address) { + f.Write(MAXWELL3D_REG_INDEX(report_semaphore.address_high), 0); + f.Write(MAXWELL3D_REG_INDEX(report_semaphore.address_low), address); + f.Write(MAXWELL3D_REG_INDEX(report_semaphore.payload), + f.Reg(MAXWELL3D_REG_INDEX(report_semaphore.payload)) + 1); + f.Write(MAXWELL3D_REG_INDEX(report_semaphore.query), 0); // long release + }; + f.DrawArrays(0, 3); + const u64 serial = FastDraw::Serial(); + FastDraw::g_delta = {}; + f.Write(0x155, f.Reg(0x155) + 0x40); // SET_MME_MEM_ADDRESS_B + report(0x30000); + CHECK(FastDraw::Serial() == serial); + REQUIRE(FastDraw::g_delta.num_writes == 1); + CHECK(FastDraw::g_delta.write_begin[0] == 0x30000); + CHECK(FastDraw::g_delta.write_end[0] == 0x30010); + f.DrawArrays(3, 3); + CHECK(f.Reusable()); + + // Into the texture headers a delta draw keeps: a full resolve. + report(0x40020); + CHECK(FastDraw::Serial() != serial); +} + +TEST_CASE("Fast draw keeps the vertex id base as a draw parameter", "[fast_draw]") { + FastDrawFixture f; + f.DrawArrays(0, 3); + f.Write(MAXWELL3D_REG_INDEX(vertex_id_base), f.Reg(MAXWELL3D_REG_INDEX(vertex_id_base)) + 3); + f.DrawArrays(3, 3); + CHECK(f.Reusable()); +} + TEST_CASE("Fast draw keeps vertex stream addresses and sizes light", "[fast_draw]") { DeltaFixture f; const auto stream = [](u32 index, u32 word) { diff --git a/src/tests/video_core/maxwell_fast_method.cpp b/src/tests/video_core/maxwell_fast_method.cpp index 65d17759fc..5b15094580 100644 --- a/src/tests/video_core/maxwell_fast_method.cpp +++ b/src/tests/video_core/maxwell_fast_method.cpp @@ -3,6 +3,7 @@ #include #include +#include #include #include #include @@ -22,11 +23,14 @@ #include "core/memory.h" #include "tests/core/system_test_context.h" #include "video_core/dirty_flags.h" +#include "video_core/engines/kepler_compute.h" #include "video_core/engines/maxwell_3d.h" #include "video_core/gpu.h" #include "video_core/gpu_split.h" +#include "video_core/macro.h" #include "video_core/memory_manager.h" -#include "video_core/nx2_async_readback.h" +#include "video_core/native_qmd.h" +#include "video_core/nx2.h" #include "video_core/renderer_null/null_rasterizer.h" namespace VideoCommon { @@ -51,26 +55,6 @@ struct GpuSplitTestAccess { split.back_thread = std::jthread([&split](std::stop_token stop) { split.BackLoop(stop); }); } - static u64 Published(const GpuSplit& split) { - return split.published.load(std::memory_order_acquire); - } - - static u64 BackIdleAt(const GpuSplit& split) { - return split.back_idle_at.load(std::memory_order_relaxed); - } - - /// The back's side as SettleIdleBack sees it, without a back thread. - static void SetBack(GpuSplit& split, bool running, bool quiet, u64 consumed) { - split.back_running.store(running); - split.back_quiet.store(quiet); - split.consumed.store(consumed); - split.hook_requested.store(false); - } - - static bool HookRequested(const GpuSplit& split) { - return split.hook_requested.load(); - } - static u64 NearWrap() { return GpuSplit::RingWords - 1; } @@ -98,6 +82,17 @@ struct GpuSplitTestAccess { return GpuSplit::WriteSlot(a >> GpuSplit::WriteGranuleBits) == GpuSplit::WriteSlot(b >> GpuSplit::WriteGranuleBits); } + + /// Runs what was sent and not run yet on the calling thread, as the back does. + static void RunBack(GpuSplit& split) { + GpuSplit::on_back_thread = true; + u64 position = split.consumed.load(); + while (position < split.write_position) { + position = Execute(split, position, split.write_position); + } + split.consumed.store(position); + GpuSplit::on_back_thread = false; + } }; } // namespace VideoCommon @@ -236,15 +231,6 @@ u32 RandomArgument(std::mt19937& rng, u32 method, const Maxwell3D& engine) { return rng() % 64; } -/// EDEN_NX2_DMA_READ_VERIFY for a test, restored after it. -struct DmaReadVerifyScope { - DmaReadVerifyScope() : previous{VideoCore::AsyncReadback::g_dma_read_verify.exchange(true)} {} - ~DmaReadVerifyScope() { - VideoCore::AsyncReadback::g_dma_read_verify = previous; - } - bool previous; -}; - /// A Turing MME instruction from its fields (TuringMacroInterpreter::Decode); a lane left alone /// adds the zero register to itself. struct Mme { @@ -830,13 +816,20 @@ TEST_CASE("Split consumption does not complete deferred downstream callbacks", REQUIRE(output == std::array{1, 2, 3}); } -TEST_CASE("Split remembers the semaphore writes its back has not run yet", +TEST_CASE("Split back runs a requested tick while it waits for work and between entries", "[video_core][gpu_split_dependencies]") { using Access = VideoCommon::GpuSplitTestAccess; Fixture f; f.system.Initialize(); VideoCommon::GpuSplit split; + std::atomic ticks{}; + std::atomic entered{}; std::atomic release{}; + u32 ticks_at_second{~0U}; + split.SetTickHook([](void* counter) { static_cast*>(counter)->fetch_add(1); }, + &ticks); + // Before the back runs, a request does nothing, nor is it kept for later. + split.RequestTick(); Access::Start(split, f.system); SCOPE_EXIT { split.Stop(); @@ -845,127 +838,36 @@ TEST_CASE("Split remembers the semaphore writes its back has not run yet", SCOPE_EXIT { release = true; }; - // Releases behind an entry the back is still in. - split.Defer([&release] { + const auto wait_until = [](const auto& done) { + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds{5}; + while (!done() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds{1}); + } + return done(); + }; + REQUIRE(wait_until([&] { return split.BackRunning(); })); + // The back has nothing to do and goes to sleep: a request wakes it and runs at once. + std::this_thread::sleep_for(std::chrono::milliseconds{20}); + REQUIRE(ticks.load() == 0); + split.RequestTick(); + REQUIRE(wait_until([&] { return ticks.load() == 1; })); + // The back runs an entry: the request runs once it ends, before the next entry. + split.Defer([&] { + entered = true; while (!release.load()) { std::this_thread::yield(); } }); - split.Defer([] {}); - split.NoteSemaphoreWrite(0x1000, 5, true); - split.Defer([] {}); - split.NoteSemaphoreWrite(0x2002, 7, false); - split.Publish(); - auto queued = split.QueuedSemaphoreWriteAt(0x1000); - REQUIRE(queued.has_value()); - REQUIRE(queued->known); - REQUIRE(queued->payload == 5); - // By 4-byte word: the second write is at 0x2000. No write was sent to 0x1004. - queued = split.QueuedSemaphoreWriteAt(0x2000); - REQUIRE(queued.has_value()); - REQUIRE_FALSE(queued->known); - REQUIRE_FALSE(split.QueuedSemaphoreWriteAt(0x1004).has_value()); - // The last write sent to a word decides. - split.Defer([] {}); - split.NoteSemaphoreWrite(0x1000, 9, true); + split.Defer([&] { ticks_at_second = ticks.load(); }); split.Publish(); - queued = split.QueuedSemaphoreWriteAt(0x1000); - REQUIRE(queued.has_value()); - REQUIRE(queued->payload == 9); - // Once the back ran them, the words hold their values in memory. + REQUIRE(wait_until([&] { return entered.load(); })); + split.RequestTick(); + std::this_thread::sleep_for(std::chrono::milliseconds{5}); + REQUIRE(ticks.load() == 1); release = true; split.Drain(VideoCommon::DrainSite::Other); - REQUIRE_FALSE(split.QueuedSemaphoreWriteAt(0x1000).has_value()); - REQUIRE_FALSE(split.QueuedSemaphoreWriteAt(0x2000).has_value()); -} - -TEST_CASE("Split publishes to a back that waits for work after a few words", - "[video_core][gpu_split_dependencies]") { - using Access = VideoCommon::GpuSplitTestAccess; - Fixture f; - f.system.Initialize(); - VideoCommon::GpuSplit split; - Access::Start(split, f.system); - SCOPE_EXIT { - split.Stop(); - }; - // Publishing raises the interval to 256 words: the words below stay with the front unless it - // feeds the waiting back. - split.Defer([] {}); - split.Publish(); - const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds{10}; - while (Access::BackIdleAt(split) != Access::End(split)) { - REQUIRE(std::chrono::steady_clock::now() < deadline); - std::this_thread::yield(); - } - const u64 published = Access::Published(split); - // An empty call takes 3 words. - split.Defer([] {}); - split.Defer([] {}); - REQUIRE(Access::Published(split) == published); - split.Defer([] {}); - REQUIRE(Access::Published(split) == Access::End(split)); -} - -TEST_CASE("Split asks a back that ran everything for its idle hook while the front polls", - "[video_core][gpu_split_dependencies]") { - using Access = VideoCommon::GpuSplitTestAccess; - Fixture f; - VideoCommon::GpuSplit split; - Access::Initialize(split, f.system); - const bool counting = VideoCommon::g_acquire_counting; - VideoCommon::g_acquire_counting = true; - SCOPE_EXIT { - Access::SetBack(split, false, true, 0); - VideoCommon::g_acquire_counting = counting; - }; - const auto settled = [&split] { - const u64 before = VideoCommon::g_acquire_stats.settles.load(); - split.SettleIdleBack(); - const bool asked = Access::HookRequested(split); - REQUIRE(asked == (VideoCommon::g_acquire_stats.settles.load() == before + 1)); - return asked; - }; - Access::SetBack(split, true, false, Access::End(split)); - // Ran everything, hook not run yet: asked once. - REQUIRE(settled()); - REQUIRE(Access::HookRequested(split)); - // Already ran its hook. - Access::SetBack(split, true, true, Access::End(split)); - REQUIRE_FALSE(settled()); - // Still has work. - Access::SetBack(split, true, false, Access::End(split)); - split.Defer([] {}); - REQUIRE_FALSE(settled()); - // Not running. - Access::SetBack(split, false, false, Access::End(split)); - REQUIRE_FALSE(settled()); -} - -TEST_CASE("Split drain waits for the back's idle hook", - "[video_core][gpu_split_dependencies]") { - using Access = VideoCommon::GpuSplitTestAccess; - Fixture f; - f.system.Initialize(); - VideoCommon::GpuSplit split; - static std::atomic hooks; - hooks = 0; - // Slow, so a drain that returned before the hook finished sees the old count. - split.SetIdleHook( - [](void*) { - std::this_thread::sleep_for(std::chrono::milliseconds{2}); - hooks.fetch_add(1); - }, - nullptr); - Access::Start(split, f.system); - SCOPE_EXIT { - split.Stop(); - }; - for (u32 i = 1; i <= 3; ++i) { - split.Defer([] {}); - split.Drain(VideoCommon::DrainSite::Acquire); - REQUIRE(hooks.load() >= i); - } + REQUIRE(ticks_at_second == 2); + REQUIRE(ticks.load() == 2); } TEST_CASE("Split command list fetches wait only for the writers before the last host sync", @@ -1049,8 +951,8 @@ TEST_CASE("Split command list fetches wait only for the writers before the last release[2] = true; split.Drain(VideoCommon::DrainSite::Other); - // Nor for other work between the last writer before the wait and the wait, when another - // writer follows the wait. + // Work between the earlier writer and the sync is not a writer, even if a later + // writer was sent after the sync. std::atomic other{}; std::atomic later{}; std::atomic other_done{}; @@ -1087,6 +989,211 @@ TEST_CASE("Split command list fetches wait only for the writers before the last split.Drain(VideoCommon::DrainSite::Other); } +TEST_CASE("Split tells whether compute or copy work it was sent is still queued", + "[video_core][gpu_split_dependencies]") { + using Access = VideoCommon::GpuSplitTestAccess; + Fixture f; + f.system.Initialize(); + VideoCommon::GpuSplit split; + std::atomic release{}; + Access::Start(split, f.system); + SCOPE_EXIT { + split.Stop(); + }; + // Runs before Stop: a failed check must not leave the back in the entry below. + SCOPE_EXIT { + release = true; + }; + REQUIRE_FALSE(split.WritersPending()); + split.Defer([&release] { + while (!release.load()) { + std::this_thread::yield(); + } + }); + split.NoteWriter(); + split.Publish(); + REQUIRE(split.WritersPending()); + release = true; + split.Drain(VideoCommon::DrainSite::Other); + REQUIRE_FALSE(split.WritersPending()); + // Other work does not count. + split.Defer([] {}); + split.Publish(); + REQUIRE_FALSE(split.WritersPending()); + split.Drain(VideoCommon::DrainSite::Other); +} + +TEST_CASE("Split remembers the semaphore writes its back has not run yet", + "[video_core][gpu_split_dependencies]") { + using Access = VideoCommon::GpuSplitTestAccess; + Fixture f; + f.system.Initialize(); + VideoCommon::GpuSplit split; + std::atomic release{}; + Access::Start(split, f.system); + SCOPE_EXIT { + split.Stop(); + }; + // Runs before Stop: a failed check must not leave the back in the entry below. + SCOPE_EXIT { + release = true; + }; + // Releases behind an entry the back is still in. + split.Defer([&release] { + while (!release.load()) { + std::this_thread::yield(); + } + }); + split.Defer([] {}); + split.NoteSemaphoreWrite(0x1000, 5, true); + split.Defer([] {}); + split.NoteSemaphoreWrite(0x2002, 7, false); + split.Publish(); + auto queued = split.QueuedSemaphoreWriteAt(0x1000); + REQUIRE(queued.has_value()); + REQUIRE(queued->known); + REQUIRE(queued->payload == 5); + // By 4-byte word: the second write is at 0x2000. No write was sent to 0x1004. + queued = split.QueuedSemaphoreWriteAt(0x2000); + REQUIRE(queued.has_value()); + REQUIRE_FALSE(queued->known); + REQUIRE_FALSE(split.QueuedSemaphoreWriteAt(0x1004).has_value()); + // The last write sent to a word decides. + split.Defer([] {}); + split.NoteSemaphoreWrite(0x1000, 9, true); + split.Publish(); + queued = split.QueuedSemaphoreWriteAt(0x1000); + REQUIRE(queued.has_value()); + REQUIRE(queued->payload == 9); + // Once the back ran them, the words hold their values in memory. + release = true; + split.Drain(VideoCommon::DrainSite::Other); + REQUIRE_FALSE(split.QueuedSemaphoreWriteAt(0x1000).has_value()); + REQUIRE_FALSE(split.QueuedSemaphoreWriteAt(0x2000).has_value()); +} + +TEST_CASE("Split counts a fenced semaphore write in memory once its fence ran, not before", + "[video_core][gpu_split_dependencies]") { + using Access = VideoCommon::GpuSplitTestAccess; + using Landing = VideoCommon::GpuSplit::Landing; + Fixture f; + f.system.Initialize(); + VideoCommon::GpuSplit split; + Access::Start(split, f.system); + SCOPE_EXIT { + split.Stop(); + }; + // A fence-flagged report lands once the fence after it ran (QueryCacheBase::CounterReport), + // which tells with the entry the back gave it. + split.Defer([] {}); + std::optional entry; + split.Defer([&split, &entry] { entry = split.BackEntry(); }); + split.NoteSemaphoreWrite(0x1000, 21, true, false, Landing::Fenced); + split.Drain(VideoCommon::DrainSite::Other); + REQUIRE(entry.has_value()); + REQUIRE(*entry > 0); + REQUIRE_FALSE(split.BackEntry().has_value()); + auto queued = split.QueuedSemaphoreWriteAt(0x1000); + REQUIRE(queued.has_value()); + REQUIRE(queued->fenced); + REQUIRE(queued->payload == 21); + // A write that lands as the back runs it, sent before that fence ran: the word holds either. + split.Defer([] {}); + split.NoteSemaphoreWrite(0x1000, 22, true); + split.Drain(VideoCommon::DrainSite::Other); + queued = split.QueuedSemaphoreWriteAt(0x1000); + REQUIRE(queued.has_value()); + REQUIRE(queued->fenced); + REQUIRE_FALSE(queued->known); + // A fence of an entry before it lands nothing of it; its own lands it. + split.NoteFencedWrite(*entry - 1); + REQUIRE(split.QueuedSemaphoreWriteAt(0x1000).has_value()); + split.NoteFencedWrite(*entry); + REQUIRE_FALSE(split.QueuedSemaphoreWriteAt(0x1000).has_value()); + // Once it landed, a write after it holds the word once the back ran it. + split.Defer([] {}); + split.NoteSemaphoreWrite(0x1000, 23, true); + queued = split.QueuedSemaphoreWriteAt(0x1000); + REQUIRE(queued.has_value()); + REQUIRE_FALSE(queued->fenced); + REQUIRE(queued->payload == 23); + split.Drain(VideoCommon::DrainSite::Other); + REQUIRE_FALSE(split.QueuedSemaphoreWriteAt(0x1000).has_value()); + // A later entry's fence lands the fenced writes of the entries before it. + split.Defer([] {}); + split.NoteSemaphoreWrite(0x2000, 24, true, false, Landing::Fenced); + std::optional later; + split.Defer([&split, &later] { later = split.BackEntry(); }); + split.Drain(VideoCommon::DrainSite::Other); + REQUIRE(split.QueuedSemaphoreWriteAt(0x2000).has_value()); + REQUIRE(later.has_value()); + split.NoteFencedWrite(*later); + REQUIRE_FALSE(split.QueuedSemaphoreWriteAt(0x2000).has_value()); +} + +TEST_CASE("Split waits for a semaphore write's position, not for the work sent after it", + "[video_core][gpu_split_dependencies]") { + using Access = VideoCommon::GpuSplitTestAccess; + Fixture f; + f.system.Initialize(); + VideoCommon::GpuSplit split; + std::array, 2> release{}; + Access::Start(split, f.system); + SCOPE_EXIT { + split.Stop(); + }; + // Runs before Stop: a failed check must not leave the back in the entries below. + SCOPE_EXIT { + release[0] = true; + release[1] = true; + }; + const auto blocking = [&split](std::atomic& flag) { + split.Defer([&flag] { + while (!flag.load()) { + std::this_thread::yield(); + } + }); + }; + // A release the front can't value (a reduction, a counter) sent by compute work, behind an + // entry the back is still in, then more work behind another. + blocking(release[0]); + split.Defer([] {}); + split.NoteSemaphoreWrite(0x1000, 0, false); + split.NoteWriter(); + const u64 release_position = split.Position(); + REQUIRE(split.LastWriterPosition() == release_position); + blocking(release[1]); + split.Publish(); + const auto queued = split.QueuedSemaphoreWriteAt(0x1000); + REQUIRE(queued.has_value()); + REQUIRE(queued->position == release_position); + REQUIRE_FALSE(split.Ran(release_position)); + std::atomic done{}; + bool waited{}; + std::thread waiter([&] { + waited = split.WaitForPosition(release_position, VideoCommon::DrainSite::Acquire); + done = true; + }); + std::this_thread::sleep_for(std::chrono::milliseconds{20}); + REQUIRE_FALSE(done.load()); + release[0] = true; + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds{5}; + while (!done.load() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds{1}); + } + REQUIRE(done.load()); + waiter.join(); + REQUIRE(waited); + // The release ran; the work after it still runs. + REQUIRE(split.Ran(release_position)); + REQUIRE_FALSE(split.QueuedSemaphoreWriteAt(0x1000).has_value()); + REQUIRE_FALSE(split.Ran(split.Position())); + release[1] = true; + split.Drain(VideoCommon::DrainSite::Other); + REQUIRE(split.Ran(split.Position())); + REQUIRE_FALSE(split.WaitForPosition(release_position, VideoCommon::DrainSite::Acquire)); +} + TEST_CASE("Split restart resets writer positions without replaying old releases", "[video_core][gpu_split_dependencies]") { using Access = VideoCommon::GpuSplitTestAccess; @@ -1101,6 +1208,7 @@ TEST_CASE("Split restart resets writer positions without replaying old releases" REQUIRE(Access::End(split) == 0); REQUIRE_FALSE(split.WaitForWriters()); REQUIRE_FALSE(split.WaitForSync()); + REQUIRE_FALSE(split.WaitForSyncedWriters()); split.Defer([&, session] { released = session; ++release_count; @@ -1181,9 +1289,6 @@ TEST_CASE("Split macro DMA reads wait only for the queued work that can write th "[video_core][gpu_split_dependencies]") { using Access = VideoCommon::GpuSplitTestAccess; using Writes = VideoCommon::GpuSplit::Writes; - auto& counters = VideoCore::AsyncReadback::g_counters; - // No per-second report resets the counters while the test reads them. - counters.window_start = VideoCore::AsyncReadback::Now(); Fixture f; f.system.Initialize(); Kernel::KProcess process{f.system.Kernel()}; @@ -1224,15 +1329,15 @@ TEST_CASE("Split macro DMA reads wait only for the queued work that can write th // Where reads wait: writes are placed by device memory, any GPU address of it. VideoCommon::GpuSplit split; Access::Initialize(split, f.system); - split.Defer([] {}, 0, Writes::Noted); + split.Defer([] {}, Writes::Noted); split.NoteWrite(f.memory, alias + 0x48, 4); const u64 noted = Access::End(split); - split.Defer([] {}, 0, Writes::Noted); + split.Defer([] {}, Writes::Noted); split.NoteWrite(f.memory, 0x300000, 16); // Unmapped: nothing is written CHECK(Access::WritesTarget(split, f.memory, word, 0x18) == noted); CHECK(Access::WritesTarget(split, f.memory, other, 0x18) == 0); // Writes too long to place, and work nobody noted the writes of, count as anywhere. - split.Defer([] {}, 0, Writes::Noted); + split.Defer([] {}, Writes::Noted); split.NoteWrite(f.memory, gpu_address + 0x1000, 0x8000); CHECK(Access::WritesTarget(split, f.memory, other, 0x18) == Access::End(split)); split.Defer([] {}); @@ -1253,7 +1358,7 @@ TEST_CASE("Split macro DMA reads wait only for the queued work that can write th SCOPE_EXIT { split.Stop(); }; - std::array, 6> release{}; + std::array, 5> release{}; std::atomic done{}; std::thread reader; SCOPE_EXIT { @@ -1343,7 +1448,7 @@ TEST_CASE("Split macro DMA reads wait only for the queued work that can write th std::this_thread::yield(); } }, - 0, writes); + writes); split.Publish(); }; @@ -1357,12 +1462,8 @@ TEST_CASE("Split macro DMA reads wait only for the queued work that can write th // Work that writes nothing and a write of other words, still queued: the read goes ahead. blocking(release[0], Writes::Noted); upload(other, 0x33333333); - u64 reads = counters.dma_reads; - u64 waited = counters.dma_waited; read_on_reader(word); REQUIRE(finishes()); - CHECK(counters.dma_reads == reads + 1); - CHECK(counters.dma_waited == waited); reader.join(); release[0] = true; split.Drain(VideoCommon::DrainSite::Other); @@ -1372,16 +1473,10 @@ TEST_CASE("Split macro DMA reads wait only for the queued work that can write th // A write of the word through its other GPU address: the read waits for it and sees it. blocking(release[1], Writes::Noted); upload(alias + 0x40, 0x44444444); - reads = counters.dma_reads; - waited = counters.dma_waited; - const u64 drained = counters.dma_drained; read_on_reader(word); REQUIRE(waits()); release[1] = true; REQUIRE(finishes()); - CHECK(counters.dma_reads == reads + 1); - CHECK(counters.dma_waited == waited + 1); - CHECK(counters.dma_drained == drained); reader.join(); split.Drain(VideoCommon::DrainSite::Other); CHECK(engine.regs.reg_array[state] == 0x44444444); @@ -1396,28 +1491,15 @@ TEST_CASE("Split macro DMA reads wait only for the queued work that can write th split.Drain(VideoCommon::DrainSite::Other); CHECK(engine.regs.reg_array[state] == 0x33333333); - // With EDEN_NX2_DMA_READ_VERIFY, a read that needn't drain drains anyway to compare its words. - { - const DmaReadVerifyScope verify; - blocking(release[3], Writes::Noted); - read_on_reader(other); - REQUIRE(waits()); - release[3] = true; - REQUIRE(finishes()); - reader.join(); - split.Drain(VideoCommon::DrainSite::Other); - CHECK(engine.regs.reg_array[state] == 0x33333333); - } - // Macro 1 runs on the back: the front goes on past that work, and the back reads the word // after the upload queued behind it. front(state, 0x77777777); - blocking(release[4], Writes::Anywhere); + blocking(release[3], Writes::Anywhere); upload(word, 0x55555555); read_on_reader(word, 0xe02); REQUIRE(finishes()); reader.join(); - release[4] = true; + release[3] = true; split.Drain(VideoCommon::DrainSite::Other); CHECK(engine.regs.reg_array[state] == 0x55555555); @@ -1429,12 +1511,12 @@ TEST_CASE("Split macro DMA reads wait only for the queued work that can write th CHECK(engine.regs.reg_array[state] == 0x77777777); // A front macro that reads the register waits for the back's macro and gets its value. - blocking(release[5], Writes::Noted); + blocking(release[4], Writes::Noted); upload(word, 0x66666666); dma_read(word, 0xe02); copy_on_reader(); REQUIRE(waits()); - release[5] = true; + release[4] = true; REQUIRE(finishes()); reader.join(); split.Drain(VideoCommon::DrainSite::Other); @@ -1453,3 +1535,209 @@ TEST_CASE("Split macro DMA reads wait only for the queued work that can write th CHECK(engine.regs.reg_array[state] == 0x88888888); CHECK(engine.regs.reg_array[copy] == 0x88888888); } + +namespace { + +// Two pages of guest memory at gpu_address (device_address) for the tests below. +struct MappedPages { + static constexpr VAddr cpu_address = 0x4000; + static constexpr DAddr device_address = 0x10000; + static constexpr GPUVAddr gpu_address = 0x100000; + + explicit MappedPages(Fixture& f) : fixture{f}, process{f.system.Kernel()} { + auto& table = process.GetPageTable().GetImpl(); + table.Resize(20, 12); + backing = f.physical.GetPointerFromRaw(0x10000); + for (u64 page = 0; page < 2; ++page) { + const VAddr address = cpu_address + page * 0x1000; + table.entries.GetAndFault(address >> 12) + .Store(false, Common::PageType::Memory, 1, + reinterpret_cast(backing + page * 0x1000) - address); + } + process.GetMemory().SetCurrentPageTable(process); + asid = f.device_memory.RegisterProcess(&process.GetMemory()); + f.device_memory.BindInterface(&f.rasterizer); + f.device_memory.Map(device_address, cpu_address, 0x2000, asid); + f.memory.BindRasterizer(&f.rasterizer); + f.memory.Map(gpu_address, device_address, 0x2000, Tegra::PTEKind::PITCH, false); + std::memset(backing, 0, 0x2000); + } + ~MappedPages() { + fixture.device_memory.Unmap(device_address, 0x2000); + fixture.device_memory.UnregisterProcess(asid); + fixture.system.Kernel().UnregisterKernelObject(&process); + } + MappedPages(const MappedPages&) = delete; + MappedPages& operator=(const MappedPages&) = delete; + + Fixture& fixture; + Kernel::KProcess process; + u8* backing{}; + Core::Asid asid{}; +}; + +} // Anonymous namespace + +TEST_CASE("Split tells guest memory writes from what only the GPU caches hold", + "[video_core][gpu_split_dependencies]") { + using Access = VideoCommon::GpuSplitTestAccess; + using Writes = VideoCommon::GpuSplit::Writes; + Fixture f; + f.system.Initialize(); + MappedPages pages{f}; + constexpr GPUVAddr word = MappedPages::gpu_address + 0x40; + constexpr GPUVAddr other = MappedPages::gpu_address + 0x140; + REQUIRE_FALSE(Access::ShareWriteSlot(MappedPages::device_address + 0x40, + MappedPages::device_address + 0x140)); + VideoCommon::GpuSplit split; + Access::Initialize(split, f.system); + split.Defer([] {}, Writes::Noted); + split.NoteWrite(f.memory, word, 4); + const u64 noted = Access::End(split); + // Shader stores: a read of the caches waits for them, a read of guest memory only for the + // writes noted there. + split.Defer([] {}, Writes::Caches); + CHECK(Access::WritesTarget(split, f.memory, other, 4) == Access::End(split)); + CHECK(split.MemoryWritesTarget(f.memory, word, 4) == noted); + CHECK(split.MemoryWritesTarget(f.memory, other, 4) == 0); + CHECK(split.LastUnplacedWrite() == 0); + // Work nobody noted the writes of writes guest memory anywhere. + split.Defer([] {}); + CHECK(split.LastUnplacedWrite() == Access::End(split)); + CHECK(split.MemoryWritesTarget(f.memory, other, 4) == Access::End(split)); + // A maximum's lower bound. + split.NoteSemaphoreWrite(MappedPages::device_address + 0x40, 5, true, true); + const auto queued = split.QueuedSemaphoreWriteAt(MappedPages::device_address + 0x40); + REQUIRE(queued.has_value()); + CHECK(queued->known); + CHECK(queued->at_least); + CHECK(queued->payload == 5); +} + +TEST_CASE("Split front values the releases of the QMD launches it sends as the back runs them", + "[video_core][gpu_split_dependencies]") { + using Access = VideoCommon::GpuSplitTestAccess; + using Descriptor = Tegra::NativeQmdScheduler::Descriptor; + using Writes = VideoCommon::GpuSplit::Writes; + constexpr u32 Enable = 1U << 23; + constexpr u32 OneWord = 1U << 30; + constexpr u32 Maximum = 1U << 19; + constexpr u32 Reduction = 1U << 26; // ADD + constexpr GPUVAddr qmds = MappedPages::gpu_address; + constexpr GPUVAddr words = MappedPages::gpu_address + 0x1000; + constexpr DAddr word_devices = MappedPages::device_address + 0x1000; + Fixture f; + f.system.Initialize(); + Tegra::Nx2ClearGpuWrittenPages(); + MappedPages pages{f}; + Tegra::Engines::KeplerCompute compute{f.memory}; + Tegra::MacroEngine macros{true}; + std::array positions{}; + compute.SetNativeMacroMode(¯os, positions); + compute.BindRasterizer(&f.rasterizer); + VideoCommon::GpuSplit split; + Access::Initialize(split, f.system); + const auto send = [&](u32 method, u32 argument) { + compute.FrontCallMethod(f.system, split, method, argument, true, nullptr); + }; + const auto make_qmd = [](u32 dependencies, u32 word_offset, u32 flags, u32 payload) { + Descriptor qmd{}; + qmd[18] = 0x30; + qmd[4] = dependencies << 14; + qmd[24] = static_cast(words) + word_offset; + qmd[25] = flags; + qmd[26] = payload; + return qmd; + }; + // An inline QMD, in memory already (the null rasterizer's inline writes write nothing): its + // words but the last, then, after the back ran them when `caught_up`, the last. + const auto inline_qmd = [&](GPUVAddr address, const Descriptor& qmd, bool caught_up) { + std::memcpy(pages.backing + (address - MappedPages::gpu_address), qmd.data(), sizeof(qmd)); + send(0xc6, static_cast((address >> 8) >> 32)); + send(0xc7, static_cast(address >> 8)); + for (u32 i = 0; i < 63; ++i) { + send(0xc8 + i, qmd[i]); + } + if (caught_up) { + Access::RunBack(split); + } + send(0x107, qmd[63]); + }; + const auto word = [&](u32 offset) { return f.memory.Read(words + offset); }; + const auto queued = [&](u32 offset) { + return split.QueuedSemaphoreWriteAt(word_devices + offset); + }; + + // A QMD that runs at once: its release is the front's once its table is the back's, which it + // takes over when the back has run everything sent. + inline_qmd(qmds, make_qmd(0, 0, Enable | OneWord, 5), true); + auto release = queued(0); + REQUIRE(release.has_value()); + CHECK(release->known); + CHECK_FALSE(release->at_least); + CHECK(release->payload == 5); + Access::RunBack(split); + + // One that waits for a dependency runs at the decrement, a maximum with the word in memory. + f.memory.Write(words + 0x10, 9); + inline_qmd(qmds + 0x100, make_qmd(1, 0x10, Enable | OneWord | Maximum, 7), false); + CHECK_FALSE(queued(0x10).has_value()); + send(0xad, static_cast((qmds + 0x100) >> 8)); + Access::RunBack(split); + send(0xb0, 7); + release = queued(0x10); + REQUIRE(release.has_value()); + CHECK(release->known); + CHECK_FALSE(release->at_least); + CHECK(release->payload == 9); + Access::RunBack(split); + CHECK(word(0x10) == 9); + + // Over a value the front can't tell, a maximum holds its payload at least. + split.Defer([] {}, Writes::Noted); + split.NoteSemaphoreWrite(word_devices + 0x10, 0, false); + inline_qmd(qmds + 0x200, make_qmd(0, 0x10, Enable | OneWord | Maximum, 12), false); + release = queued(0x10); + REQUIRE(release.has_value()); + CHECK(release->known); + CHECK(release->at_least); + CHECK(release->payload == 12); + Access::RunBack(split); + CHECK(word(0x10) == 12); + + // A reduction over the release before it, still queued. + inline_qmd(qmds + 0x300, make_qmd(0, 0x20, Enable | OneWord | Maximum, 4), false); + inline_qmd(qmds + 0x400, make_qmd(0, 0x20, Enable | OneWord | Reduction, 3), false); + release = queued(0x20); + REQUIRE(release.has_value()); + CHECK(release->known); + CHECK_FALSE(release->at_least); + CHECK(release->payload == 7); + Access::RunBack(split); + CHECK(word(0x20) == 7); + + // Work that may write guest memory anywhere and change the mappings, still queued: the front + // can't tell what the back reads yet, the launch waits for it. + split.Defer([] {}); + inline_qmd(qmds + 0x500, make_qmd(0, 0x30, Enable | OneWord, 1), false); + CHECK_FALSE(queued(0x30).has_value()); + Access::RunBack(split); + // The back ran everything: the front takes its table over, the waiting launch with it. + inline_qmd(qmds + 0x600, make_qmd(0, 0x30, Enable | OneWord, 2), true); + release = queued(0x30); + REQUIRE(release.has_value()); + CHECK(release->payload == 2); + Access::RunBack(split); + + // A write to a waiting QMD still queued when its decrement is sent: the launch waits, not + // valued then. + inline_qmd(qmds + 0x700, make_qmd(1, 0x40, Enable | OneWord | Maximum, 3), false); + Access::RunBack(split); + split.Defer([] {}, Writes::Noted); + split.NoteWrite(f.memory, qmds + 0x700 + 26 * sizeof(u32), sizeof(u32)); + send(0xad, static_cast((qmds + 0x700) >> 8)); + send(0xb0, 7); + CHECK_FALSE(queued(0x40).has_value()); + Access::RunBack(split); + CHECK(word(0x40) == 3); +} diff --git a/src/tests/video_core/native_qmd.cpp b/src/tests/video_core/native_qmd.cpp index 560ae4b13f..2a7943a018 100644 --- a/src/tests/video_core/native_qmd.cpp +++ b/src/tests/video_core/native_qmd.cpp @@ -4,7 +4,9 @@ #include #include #include +#include #include +#include #include #include @@ -149,3 +151,141 @@ TEST_CASE("Semaphore reductions follow the documented ops", "[video_core][native REQUIRE(ReduceSemaphore(6, false, 0b1100, 0b1010) == 0b1110); REQUIRE(ReduceSemaphore(7, false, 0b1100, 0b1010) == 0b0110); } + +TEST_CASE("A QMD scheduler with a table of its own schedules apart from the shared one", + "[video_core][native_qmd]") { + constexpr Address address = 0x40000; + auto qmd = MakeQmd(); + qmd[4] = 1U << 14; // One dependency + SetRelease(qmd, 24, 0x5000, Enable | OneWord, 1); + const std::map memory{{address, qmd}}; + const auto signal = [&](NativeQmdScheduler& scheduler, Word action) { + std::vector released; + const auto error = scheduler.Signal( + address, action, + [&](Address qmd_address, Descriptor& descriptor) { + const auto it = memory.find(qmd_address); + if (it == memory.end()) { + return false; + } + descriptor = it->second; + return true; + }, + [&](Address destination, Word payload, Word flags) { + released.push_back({destination, payload, flags}); + return true; + }); + REQUIRE(error.empty()); + return released; + }; + const std::vector run{{0x5000, 1, Enable | OneWord}}; + // Loaded on the shared table, it waits for its dependency. + NativeQmdScheduler shared; + REQUIRE(signal(shared, 3).empty()); + // A copy of the table runs it at the decrement as the shared one does, and apart from it. + NativeQmdScheduler own{NativeQmdScheduler::OwnTable{}}; + own.CopyStates(shared); + REQUIRE(signal(own, 7) == run); + REQUIRE(signal(own, 7).empty()); // Completed: a decrement is kept for its next load + REQUIRE(signal(shared, 7) == run); + // A table that never saw the load keeps the decrement for it. + NativeQmdScheduler other{NativeQmdScheduler::OwnTable{}}; + REQUIRE(signal(other, 7).empty()); + REQUIRE(signal(other, 3) == run); +} + +TEST_CASE("A QMD scheduler takes back what an attempt changed and keeps what one committed", + "[video_core][native_qmd]") { + constexpr Address first = 0x40000; + constexpr Address second = 0x40100; + auto waiting = MakeQmd(); + waiting[4] = 1U << 14; // One dependency + SetRelease(waiting, 24, 0x5000, Enable | OneWord, 1); + auto chained = MakeQmd(); // Decrements `first` when it runs + SetRelease(chained, 24, 0x5004, Enable | OneWord, 2); + chained[15] = static_cast(first >> 8); + chained[16] = 1 | (4U << 1); + std::map memory{{first, waiting}, {second, chained}}; + bool unreadable{}; + const auto signal = [&](NativeQmdScheduler& scheduler, Address address, Word action) { + std::vector released; + const auto error = scheduler.Signal( + address, action, + [&](Address qmd_address, Descriptor& descriptor) { + const auto it = memory.find(qmd_address); + if (it == memory.end() || (unreadable && qmd_address == first)) { + return false; + } + descriptor = it->second; + return true; + }, + [&](Address destination, Word payload, Word flags) { + released.push_back({destination, payload, flags}); + return true; + }); + return std::make_pair(std::string{error}, released); + }; + NativeQmdScheduler own{NativeQmdScheduler::OwnTable{}}; + NativeQmdScheduler reference{NativeQmdScheduler::OwnTable{}}; + REQUIRE(signal(own, first, 3).second.empty()); // Loaded, waits for its dependency + REQUIRE(signal(reference, first, 3).second.empty()); + // The chained launch runs and decrements `first`, which can't be read: taken back, the table + // is as before, `second` never ran. + unreadable = true; + own.BeginAttempt(); + REQUIRE_FALSE(signal(own, second, 3).first.empty()); + own.RollbackAttempt(); + REQUIRE(own.SameStates(reference)); + REQUIRE(own.StateCount() == 1); + // Tried again once it can be read: both run, as on a table that never tried. + unreadable = false; + own.BeginAttempt(); + const auto ran = signal(own, second, 3); + own.CommitAttempt(); + const std::vector both{{0x5004, 2, Enable | OneWord}, {0x5000, 1, Enable | OneWord}}; + REQUIRE(ran.first.empty()); + REQUIRE(ran.second == both); + REQUIRE(signal(reference, second, 3).second == both); + REQUIRE(own.SameStates(reference)); + // A rollback after the commit changes nothing. + own.RollbackAttempt(); + REQUIRE(own.SameStates(reference)); +} + +TEST_CASE("A QMD scheduler takes back the history an attempt dropped at its limit", + "[video_core][native_qmd]") { + // Any address holds a QMD that runs at once; one of them can't be read. + const Descriptor qmd = MakeQmd(); + Address unreadable = 0; + const auto signal = [&](NativeQmdScheduler& scheduler, Address address) { + return std::string{scheduler.Signal( + address, 3, + [&](Address qmd_address, Descriptor& descriptor) { + descriptor = qmd; + return qmd_address != unreadable; + }, + [](Address, Word, Word) { return true; })}; + }; + NativeQmdScheduler own{NativeQmdScheduler::OwnTable{}}; + NativeQmdScheduler reference{NativeQmdScheduler::OwnTable{}}; + for (Address i = 0; i < 4096; ++i) { + REQUIRE(signal(own, 0x100000 + i * 0x100).empty()); + REQUIRE(signal(reference, 0x100000 + i * 0x100).empty()); + } + REQUIRE(own.StateCount() == 4096); + // A new QMD past the limit drops the finished ones first; it can't be read: all taken back. + unreadable = 0x900000; + own.BeginAttempt(); + REQUIRE_FALSE(signal(own, unreadable).empty()); + own.RollbackAttempt(); + REQUIRE(own.StateCount() == 4096); + REQUIRE(own.SameStates(reference)); + // Kept, it drops them as the reference does. + unreadable = 0; + own.BeginAttempt(); + REQUIRE(signal(own, 0x900000).empty()); + own.CommitAttempt(); + REQUIRE(signal(reference, 0x900000).empty()); + REQUIRE(own.StateCount() == 1); + REQUIRE(own.SameStates(reference)); +} diff --git a/src/tests/video_core/sampled_views.cpp b/src/tests/video_core/sampled_views.cpp index d5df6a6cfe..cee2fb138a 100644 --- a/src/tests/video_core/sampled_views.cpp +++ b/src/tests/video_core/sampled_views.cpp @@ -94,6 +94,17 @@ TEST_CASE("Buffer views and empty ids are never prepared", "[video_core][texture CHECK_FALSE(cache.Change({{buffer}, {ImageViewId{}}})); } +TEST_CASE("Empty descriptor views with their own ids never prepare the null image", + "[video_core][texture_cache]") { + Cache cache; + const ImageViewId empty = cache.views.insert(VideoCommon::NullImageViewParams{}); + REQUIRE(empty != VideoCommon::NULL_IMAGE_VIEW_ID); + REQUIRE_FALSE(cache.views[empty].IsBuffer()); + REQUIRE(cache.views[empty].image_id == VideoCommon::NULL_IMAGE_ID); + CHECK_FALSE(VideoCommon::PreparesImage(cache.views, empty)); + CHECK_FALSE(cache.Change({{empty}, {cache.AddView(cache.AddImage())}})); +} + TEST_CASE("A sampled image with an upload pending changes", "[video_core][texture_cache]") { Cache cache; const ImageId image = cache.AddImage(); diff --git a/src/tests/video_core/semaphore_waiters.cpp b/src/tests/video_core/semaphore_waiters.cpp index f33b8dac3e..f4682f7eb0 100644 --- a/src/tests/video_core/semaphore_waiters.cpp +++ b/src/tests/video_core/semaphore_waiters.cpp @@ -54,17 +54,17 @@ TEST_CASE("Semaphore waiters latch on the writes that meet them", "[video_core][ REQUIRE_FALSE(waiters.Latched(&mask)); } -TEST_CASE("Semaphore waiters refuse owners past the table and reuse freed slots", +TEST_CASE("Semaphore waiters retain more than 64 owners and replace existing owners", "[video_core][nx2_waiters]") { Nx2SemaphoreWaiters waiters; - std::array owners{}; - for (size_t i = 0; i < Nx2SemaphoreWaiters::MaxWaiters; ++i) { + std::array owners{}; + for (size_t i = 0; i + 1 < owners.size(); ++i) { REQUIRE(waiters.Add(&owners[i], 0x1000 + i * 4, 1, Nx2SemaphoreWaiters::AcquireGequal)); } const int* const extra = &owners.back(); - REQUIRE_FALSE(waiters.Add(extra, 0x8000, 1, Nx2SemaphoreWaiters::AcquireGequal)); + REQUIRE(waiters.Add(extra, 0x8000, 1, Nx2SemaphoreWaiters::AcquireGequal)); waiters.OnWrite(0x8000, 1); - REQUIRE_FALSE(waiters.Latched(extra)); + REQUIRE(waiters.Latched(extra)); // An owner already in the table moves to its new word. REQUIRE(waiters.Add(&owners[3], 0x9000, 2, Nx2SemaphoreWaiters::AcquireGequal)); waiters.OnWrite(0x100c, 1); diff --git a/src/tests/video_core/uniform_stream_reuse.cpp b/src/tests/video_core/uniform_stream_reuse.cpp index 6059b70eeb..567b3bb7c6 100644 --- a/src/tests/video_core/uniform_stream_reuse.cpp +++ b/src/tests/video_core/uniform_stream_reuse.cpp @@ -19,9 +19,21 @@ void Check(bool condition, const char* message) { struct Memory { std::array bytes{}; + u32 gathered{}; ///< Reads through ReadBlockUnsafe + u32 in_place{}; ///< Ranges GetSpan handed out void ReadBlockUnsafe(u64 address, void* destination, u32 size) { Check(address + size <= bytes.size(), "read outside guest memory"); std::memcpy(destination, bytes.data() + address, size); + ++gathered; + } + /// As the device memory manager: a range in one piece of host memory, here one that doesn't + /// cross a 4 KiB page (as if the pages after it were mapped elsewhere). + u8* GetSpan(u64 address, size_t size) { + if (address / 4096 != (address + size - 1) / 4096) { + return nullptr; + } + ++in_place; + return bytes.data() + address; } }; @@ -113,4 +125,45 @@ TEST_CASE("Uniform stream reuse preserves data through mutation and session rese bind(stage, slot, address, size); bind(stage, slot, address, size); } + Check(memory.gathered > 1000 && memory.in_place > 1000, + "the reads did not take both the gathered and the in-place path"); +} + +TEST_CASE("Uniform stream keeps require the same range and live allocation", "[video_core]") { + using VideoCommon::BindSmallUniformReusing; + using VideoCommon::RebindKeptUniform; + Memory memory; + Runtime runtime; + VideoCommon::ResetUniformStreamReuse(); + CHECK_FALSE(RebindKeptUniform(runtime, 0, 0, 0, 64)); + REQUIRE(BindSmallUniformReusing(runtime, memory, 0, 0, 0, 0, 64)); + const u32 gathered = memory.gathered; + const u32 in_place = memory.in_place; + REQUIRE(RebindKeptUniform(runtime, 0, 0, 0, 64)); + CheckBound(runtime, memory, 0, 64); + CHECK(memory.gathered == gathered); + CHECK(memory.in_place == in_place); + CHECK_FALSE(RebindKeptUniform(runtime, 0, 0, 64, 64)); + CHECK_FALSE(RebindKeptUniform(runtime, 0, 0, 0, 32)); + CHECK_FALSE(RebindKeptUniform(runtime, 0, 1, 0, 64)); + CHECK_FALSE(RebindKeptUniform(runtime, 1, 0, 0, 64)); + CHECK_FALSE(RebindKeptUniform(runtime, 5, 0, 0, 64)); + CHECK_FALSE(RebindKeptUniform(runtime, 0, 18, 0, 64)); + CHECK_FALSE(RebindKeptUniform(runtime, 0, 0, 0, 0)); + CHECK_FALSE(RebindKeptUniform(runtime, 0, 0, 0, 4097)); + + // Equal bytes at a different address reuse the allocation, but update the keep's range. + REQUIRE(BindSmallUniformReusing(runtime, memory, 0, 0, 0, 64, 64)); + CHECK(runtime.allocations.size() == 1); + CHECK_FALSE(RebindKeptUniform(runtime, 0, 0, 0, 64)); + REQUIRE(RebindKeptUniform(runtime, 0, 0, 64, 64)); + CHECK(VideoCommon::KeptUniformHolds(memory, 0, 0, 64, 64)); + memory.bytes[67] = 9; + CHECK_FALSE(VideoCommon::KeptUniformHolds(memory, 0, 0, 64, 64)); + + runtime.NextTick(); + CHECK_FALSE(RebindKeptUniform(runtime, 0, 0, 64, 64)); + REQUIRE(BindSmallUniformReusing(runtime, memory, 0, 0, 0, 64, 64)); + VideoCommon::ResetUniformStreamReuse(); + CHECK_FALSE(RebindKeptUniform(runtime, 0, 0, 64, 64)); } diff --git a/src/tests/video_core/upload_order.cpp b/src/tests/video_core/upload_order.cpp new file mode 100644 index 0000000000..4e438eb11a --- /dev/null +++ b/src/tests/video_core/upload_order.cpp @@ -0,0 +1,192 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#include +#include +#include +#include +#include + +#include "video_core/buffer_cache/upload_order.h" + +namespace { + +struct Copy { + u64 dst_offset; + u64 size; +}; + +/// The tick of each byte's last use, and every tick that used any byte. +struct ByteModel { + explicit ByteModel(size_t size) : last(size, 0) {} + + void Track(u64 offset, u64 size, u64 tick) { + for (u64 i = offset; i < offset + size; ++i) { + last[i] = tick; + } + if (size != 0) { + ticks.push_back(tick); + } + } + + bool IsUsed(u64 offset, u64 size, u64 tick) const { + for (u64 i = offset; i < offset + size; ++i) { + if (last[i] == tick) { + return true; + } + } + return false; + } + + /// The newest tick before `tick` that used a byte of [offset, offset + size). + u64 NewestByteUseBefore(u64 offset, u64 size, u64 tick) const { + u64 newest = 0; + for (u64 i = offset; i < offset + size; ++i) { + if (last[i] < tick) { + newest = (std::max)(newest, last[i]); + } + } + return newest; + } + + u64 NewestTickBefore(u64 tick) const { + u64 newest = 0; + for (const u64 used : ticks) { + if (used < tick) { + newest = (std::max)(newest, used); + } + } + return newest; + } + + std::vector last; + std::vector ticks; +}; + +} // Anonymous namespace + +TEST_CASE("Submission uses keep the newest submission's byte ranges", "[video_core]") { + VideoCommon::SubmissionUses uses; + REQUIRE_FALSE(uses.IsUsed(0, 16, 1)); + REQUIRE(uses.NewestTickBefore(1) == 0); + + uses.Track(256, 64, 5); + uses.Track(320, 64, 5); // touches the end of one: merged + uses.Track(1024, 16, 5); + uses.Track(1008, 16, 5); // touches the start of one: merged + REQUIRE(uses.RangeCount() == 2); + REQUIRE(uses.IsUsed(256, 1, 5)); + REQUIRE(uses.IsUsed(383, 1, 5)); + REQUIRE_FALSE(uses.IsUsed(384, 624, 5)); // the gap between the ranges + REQUIRE(uses.IsUsed(384, 625, 5)); + REQUIRE_FALSE(uses.IsUsed(192, 64, 5)); // ends where a range begins + REQUIRE(uses.IsUsed(192, 65, 5)); + REQUIRE(uses.IsUsed(1000, 100, 5)); + REQUIRE_FALSE(uses.IsUsed(992, 16, 5)); + REQUIRE_FALSE(uses.IsUsed(1040, 16, 5)); + REQUIRE_FALSE(uses.IsUsed(256, 64, 4)); // another submission's + REQUIRE_FALSE(uses.IsUsed(256, 0, 5)); + REQUIRE(uses.NewestTickBefore(5) == 0); + REQUIRE(uses.NewestTickBefore(6) == 5); + + // A range covering several merges them. + uses.Track(300, 800, 5); + REQUIRE(uses.RangeCount() == 1); + REQUIRE(uses.IsUsed(1050, 4, 5)); + + // The next submission starts empty; the one before is the newest earlier use. + uses.Track(4096, 4, 7); + REQUIRE(uses.RangeCount() == 1); + REQUIRE_FALSE(uses.IsUsed(256, 64, 7)); + REQUIRE(uses.IsUsed(4096, 4, 7)); + REQUIRE(uses.NewestTickBefore(7) == 5); + REQUIRE(uses.NewestTickBefore(8) == 7); +} + +TEST_CASE("Submission uses match a byte model", "[video_core]") { + constexpr u64 Size = 16384; + std::mt19937 random{0x0de7}; + for (u32 run = 0; run < 24; ++run) { + VideoCommon::SubmissionUses uses; + ByteModel model{Size}; + u64 tick = 1; + for (u32 step = 0; step < 3000; ++step) { + const u32 roll = random() % 100; + if (roll < 3) { + tick += 1 + random() % 3; + } + const u64 size = random() % 8 == 0 ? random() % 4096 : 1 + random() % 96; + const u64 offset = random() % (Size - size); + if (roll < 60) { + uses.Track(offset, size, tick); + model.Track(offset, size, tick); + continue; + } + // Ask about the submission being recorded, as an upload does. + REQUIRE(uses.IsUsed(offset, size, tick) == model.IsUsed(offset, size, tick)); + const u64 bound = uses.NewestTickBefore(tick); + REQUIRE(bound == model.NewestTickBefore(tick)); + REQUIRE(bound >= model.NewestByteUseBefore(offset, size, tick)); + const VideoCommon::UploadOrder order = + VideoCommon::DecideUploadOrder(uses, std::array{Copy{offset, size}}, tick); + REQUIRE(order.reorder == !model.IsUsed(offset, size, tick)); + if (order.reorder) { + REQUIRE(order.prior_tick == bound); + } + } + } +} + +TEST_CASE("Submission uses only grow past their range limit", "[video_core]") { + constexpr u64 Size = (VideoCommon::SubmissionUses::MaxRanges + 64) * 64; + VideoCommon::SubmissionUses uses; + ByteModel model{Size}; + std::mt19937 random{0x51de}; + // Disjoint 16-byte ranges in a random order, more of them than the limit. + std::vector slots(Size / 64); + for (u64 i = 0; i < slots.size(); ++i) { + slots[i] = i * 64; + } + std::shuffle(slots.begin(), slots.end(), random); + for (const u64 offset : slots) { + uses.Track(offset, 16, 3); + model.Track(offset, 16, 3); + REQUIRE(uses.RangeCount() <= VideoCommon::SubmissionUses::MaxRanges); + } + // Every used byte still is; unused ones may be counted as used, never the reverse. + for (u64 offset = 0; offset < Size; offset += 8) { + if (model.IsUsed(offset, 8, 3)) { + REQUIRE(uses.IsUsed(offset, 8, 3)); + } + } + REQUIRE_FALSE(uses.IsUsed(Size, 64, 3)); +} + +TEST_CASE("Uploads stay in order behind uses in their submission", "[video_core]") { + VideoCommon::SubmissionUses uses; + // Submission 4 drew from bytes 0..256. + uses.Track(0, 256, 4); + // Submission 5: the first upload goes ahead, after submission 4's work. + const auto first = VideoCommon::DecideUploadOrder(uses, std::array{Copy{128, 64}}, 5); + REQUIRE(first.reorder); + REQUIRE(first.prior_tick == 4); + uses.Track(128, 64, 5); + // A draw then reads other bytes; an upload of those must stay behind it. + uses.Track(512, 64, 5); + REQUIRE_FALSE(VideoCommon::DecideUploadOrder(uses, std::array{Copy{540, 4}}, 5).reorder); + // An upload of the first upload's bytes stays behind it too. + REQUIRE_FALSE(VideoCommon::DecideUploadOrder(uses, std::array{Copy{160, 8}}, 5).reorder); + // One copy of several is enough. + REQUIRE_FALSE(VideoCommon::DecideUploadOrder( + uses, std::array{Copy{1024, 8}, Copy{560, 8}, Copy{2048, 8}}, 5) + .reorder); + // Untouched bytes go ahead. + const auto other = VideoCommon::DecideUploadOrder(uses, std::array{Copy{1024, 8}}, 5); + REQUIRE(other.reorder); + REQUIRE(other.prior_tick == 4); + // A buffer no work used: ahead, after nothing. + const VideoCommon::SubmissionUses unused; + const auto fresh = VideoCommon::DecideUploadOrder(unused, std::array{Copy{0, 4}}, 5); + REQUIRE(fresh.reorder); + REQUIRE(fresh.prior_tick == 0); +} diff --git a/src/video_core/CMakeLists.txt b/src/video_core/CMakeLists.txt index 53cb647da1..796fa0c9e5 100644 --- a/src/video_core/CMakeLists.txt +++ b/src/video_core/CMakeLists.txt @@ -19,6 +19,7 @@ add_library(video_core STATIC buffer_cache/buffer_cache.cpp buffer_cache/buffer_cache.h buffer_cache/memory_tracker_base.h + buffer_cache/upload_order.h buffer_cache/usage_tracker.h buffer_cache/virtual_range_cache.h buffer_cache/word_manager.h @@ -100,6 +101,7 @@ add_library(video_core STATIC macro_memo.h macro_turing.h fence_manager.h + front_qmd.h gpu.cpp gpu.h gpu_timestamp.h @@ -107,6 +109,7 @@ add_library(video_core STATIC gpu_split.h gpu_thread.cpp gpu_thread.h + gpu_wait_stats.h guest_memory.h invalidation_accumulator.h memory_manager.cpp @@ -120,14 +123,19 @@ add_library(video_core STATIC nx2_cmdpage_diag.h nx2_debug_control.cpp nx2_debug_control.h + nx2_drain_check.h + nx2_entry_check.h nx2_fast_draw.cpp nx2_fast_draw.h + nx2_list_indirect.h + nx2_list_indirect_stats.h + nx2_list_stats.h nx2_profile.h nx2_stutter_trace.cpp nx2_stutter_trace.h nx2_perf.h - nx2_async_readback.h nx2_download_trace.h + nx2_read_trace.h pending_maps.h present.h pte_kind.h @@ -178,6 +186,8 @@ add_library(video_core STATIC renderer_vulkan/framebuffer_limits.h renderer_vulkan/maxwell_to_vk.cpp renderer_vulkan/maxwell_to_vk.h + renderer_vulkan/nx2_list_slots.cpp + renderer_vulkan/nx2_list_slots.h renderer_vulkan/pipeline_build.h renderer_vulkan/pipeline_build_trace.h renderer_vulkan/pipeline_helper.h @@ -219,6 +229,8 @@ add_library(video_core STATIC renderer_vulkan/vk_fence_manager.h renderer_vulkan/vk_frame_dump.cpp renderer_vulkan/vk_frame_dump.h + renderer_vulkan/vk_gpu_cost.cpp + renderer_vulkan/vk_gpu_cost.h renderer_vulkan/vk_graphics_pipeline.cpp renderer_vulkan/vk_graphics_pipeline.h renderer_vulkan/vk_multi_range_buffer.cpp @@ -240,6 +252,7 @@ add_library(video_core STATIC renderer_vulkan/vk_rasterizer.cpp renderer_vulkan/vk_rasterizer.h renderer_vulkan/submission_policy.h + renderer_vulkan/upload_batch.h renderer_vulkan/vk_render_pass_cache.cpp renderer_vulkan/vk_render_pass_cache.h renderer_vulkan/vk_resource_pool.cpp @@ -262,6 +275,8 @@ add_library(video_core STATIC renderer_vulkan/vk_turbo_mode.h renderer_vulkan/vk_update_descriptor.cpp renderer_vulkan/vk_update_descriptor.h + renderer_vulkan/vk_upload_reorder_check.cpp + renderer_vulkan/vk_upload_reorder_check.h renderer_vulkan/descriptor_payload_growth.h vulkan_common/vulkan_debug_callback.cpp vulkan_common/vulkan_debug_callback.h diff --git a/src/video_core/buffer_cache/buffer_cache.h b/src/video_core/buffer_cache/buffer_cache.h index dcb049da8b..cda8259d8d 100644 --- a/src/video_core/buffer_cache/buffer_cache.h +++ b/src/video_core/buffer_cache/buffer_cache.h @@ -11,7 +11,6 @@ #include #include #include -#include #include #include #include @@ -24,11 +23,11 @@ #include "video_core/buffer_cache/buffer_cache_base.h" #include "video_core/buffer_cache/multi_range_refresh.h" #include "video_core/buffer_cache/uniform_stream_reuse.h" -#include "video_core/gpu_split.h" #include "video_core/guest_memory.h" #include "video_core/host1x/gpu_device_memory_manager.h" #include "video_core/nx2.h" #include "video_core/nx2_cmdpage_diag.h" +#include "video_core/nx2_list_stats.h" #include "video_core/nx2_perf.h" #include "video_core/nx2_profile.h" #include "video_core/texture_cache/util.h" @@ -212,6 +211,9 @@ void BufferCache

::TickFrame() { runtime.FreeDeferredStagingBuffer(buffer); } async_buffers_death_ring.clear(); + if constexpr (HAS_READ_TRACE) { + ReadTrace::Tick(); + } } template @@ -225,14 +227,15 @@ template void BufferCache

::WriteMemory(DAddr device_addr, u64 size, bool keep_page_gpu_data) { Nx2NoteListHostWrite(device_addr, size); NoteUniformWrite(device_addr, size); - // EDEN_NX2_READ_CHECK: a newer value where an early download landed, as ClearDownload + // TEST (EDEN_NX2_READ_CHECK): a newer value where an early download landed, as ClearDownload // forgets one over GPU-modified bytes: without early downloads the write had taken the bytes // out of the GPU-modified ones, and the read's synchronous flush leaves them. Landed bytes // are no longer GPU-modified, so the write reaches no ClearDownload: written back to the // landed value after an upload of another, they read as GPU bytes the download missed. - if (!early_landed.empty()) { + if (!early_landed.empty()) [[unlikely]] { ForgetEarlyLanded(device_addr, size); } + NoteReadCheckEvent(device_addr, size, ReadCheckEvent::GuestWrite); // TEST // The written range lies within its pages: when they hold no GPU-written data, both lookups // below find nothing, so one lookup answers them. if (keep_page_gpu_data) { @@ -262,14 +265,10 @@ void BufferCache

::WriteMemory(DAddr device_addr, u64 size, bool keep_page_gpu // structures next to the draw arguments its compute dispatches fill in; the draws then read // leftover bytes as vertex counts. Download those bytes first (the CPU's own were subtracted // above, so they aren't overwritten). - // Only the scheduler's owner may download (it records and submits): unmaps and other writes - // reach this from CPU and service threads too, and submitting there raced the GPU thread's - // own submissions (crashes in Scheduler::SubmitExecution). Those keep the old behaviour; - // guest CPU writes come through TryCacheInvalidation, which sends such pages to the GPU thread. const DAddr page_begin = Common::AlignDown(device_addr, DEVICE_PAGESIZE); const DAddr page_end = Common::AlignUp(device_addr + size, DEVICE_PAGESIZE); - if (keep_page_gpu_data && - VideoCommon::GpuSplit::OwnsScheduler() && + // Unmaps and service-thread writes must not record into the GPU thread's scheduler. + if (keep_page_gpu_data && VideoCommon::GpuSplit::OwnsScheduler() && gpu_modified_ranges.Intersects(page_begin, page_end - page_begin)) { // Switch 2 titles: upload the CPU's bytes into the buffer instead, as CachedWriteMemory // does, so the page stays GPU-owned and nothing waits for the host GPU. @@ -346,7 +345,7 @@ std::optional BufferCache

::GetFlushArea(DA area->preemtive = !IsRegionGpuModified(device_addr_start_aligned, device_addr_end_aligned - device_addr_start_aligned); NoteReadBack(device_addr_start_aligned, device_addr_end_aligned, !area->preemtive); - // EDEN_NX2_READ_CHECK: a read an early download serves takes a flush, which compares + // TEST (EDEN_NX2_READ_CHECK): a read an early download serves takes a flush, which compares // what it finds with the GPU's bytes now. const bool check_early = ReadChecks() && EarlyDownloadServes(device_addr_start_aligned, @@ -358,11 +357,26 @@ std::optional BufferCache

::GetFlushArea(DA // The core took the trap, as below. read_trapped_ranges.Subtract(device_addr_start_aligned, device_addr_end_aligned - device_addr_start_aligned); + // TEST (EDEN_NX2_READ_TRACE): served by an early download. + bool early_cover = false; + if (ReadTrace::Enabled()) [[unlikely]] { + early_cover = std::ranges::any_of(read_downloads, [&](const ReadDownload& download) { + return download.early && download.write_ranges.Intersects( + device_addr_start_aligned, + device_addr_end_aligned - device_addr_start_aligned); + }); + } if (!WriteFinishedReadDownloads(device_addr_start_aligned, device_addr_end_aligned - device_addr_start_aligned)) { + if (early_cover) [[unlikely]] { + ReadTrace::Count(ReadTrace::g_early.served_in_flight); + } area->in_flight = true; return area; } + if (early_cover) [[unlikely]] { + ReadTrace::Count(ReadTrace::g_early.served_at_read); + } // The host GPU had run them: in guest memory now, nothing is left to wait for. area->preemtive = !IsRegionGpuModified(device_addr_start_aligned, device_addr_end_aligned - device_addr_start_aligned); @@ -370,8 +384,9 @@ std::optional BufferCache

::GetFlushArea(DA return area; } } - if (check_early) { + if (check_early) [[unlikely]] { area->preemtive = false; + ++read_check.early_served; } // A preflushable mark persists and downloads later GPU writes at every fence. Native titles // instead keep read traps and flush on demand. @@ -479,6 +494,12 @@ void BufferCache

::DownloadRangesMemory(std::span> if (staged.empty() && mapped.empty()) { return; } + for (const Download& download : staged) { + GpuWaitStats::AddDownloadBytes(download.copy.size); + } + for (const Download& download : mapped) { + GpuWaitStats::AddDownloadBytes(download.copy.size); + } // Once the host GPU ran the copies: the CPU writes collected meanwhile keep their bytes // too. const auto land = [&](const Async_Buffer* staging) { @@ -569,6 +590,7 @@ void BufferCache

::RecordReadDownload(std::span> r boost::container::small_vector copies; ReadDownload download; download.early = early; + download.events_before = read_check_event_count; // TEST u64 total_size_bytes = 0; const auto add_copy = [&](BufferId buffer_id, DAddr buffer_addr, DAddr begin, DAddr end) { copies.push_back({buffer_id, BufferCopy{ @@ -581,20 +603,14 @@ void BufferCache

::RecordReadDownload(std::span> r download.clean_ranges.Add(begin, end - begin); read_download_ranges.Add(begin, end - begin); }; - // EDEN_NX2_READ_CHECK: what a synchronous flush (DownloadRangesMemory) copies for + // TEST (EDEN_NX2_READ_CHECK): what a synchronous flush (DownloadRangesMemory) copies for // the read itself. std::unique_ptr check; boost::container::small_vector check_copies; u64 check_size = 0; const u64 check_every = read_check.every != 0 ? read_check.every : ReadCheckEvery(); - if (check_every != 0 && read_check.next++ % check_every == 0) { + if (check_every != 0 && read_check.next++ % check_every == 0) [[unlikely]] { check = std::make_unique(); - check->read_begin = ~DAddr{0}; - for (const auto& [own_addr, own_size] : own) { - check->read_begin = (std::min)(check->read_begin, own_addr); - check->read_end = (std::max)(check->read_end, own_addr + own_size); - } - check->early = early; for (const auto& [own_addr, own_size] : own) { ForEachBufferInRange(own_addr, own_size, [&](BufferId buffer_id, Buffer& buffer) { const DAddr buffer_addr = buffer.CpuAddr(); @@ -659,7 +675,7 @@ void BufferCache

::RecordReadDownload(std::span> r }); }); } - if (check) { + if (check) [[unlikely]] { // Each byte a synchronous flush copies is in this download or an earlier one (kept). for (const Copy& entry : check_copies) { const DAddr start = slot_buffers[entry.buffer_id].CpuAddr() + entry.copy.src_offset; @@ -691,15 +707,24 @@ void BufferCache

::RecordReadDownload(std::span> r }); } } - // EDEN_NX2_READ_CHECK: what early downloads brought to the read's pages before it + // TEST (EDEN_NX2_READ_CHECK): what early downloads brought to the read's pages before it // came. No longer GPU-modified, they are compared as they landed. struct LandedCopy { u64 offset; ///< In the check's staging, past its own offset std::vector as_landed; }; boost::container::small_vector landed_copies; - if (check && !early && !early_landed.empty()) { - const auto add_landed = [&](DAddr lower, DAddr upper, const u8* as_landed) { + if (check && !early && !early_landed.empty()) [[unlikely]] { + const auto add_landed = [&](DAddr lower, DAddr upper, const u8* as_landed, + const EarlyLanded& entry) { + for (const auto& from : entry.from) { + if (from.begin < upper && lower < from.end) { + auto origin = from; + origin.begin = (std::max)(from.begin, lower); + origin.end = (std::min)(from.end, upper); + check->origins.push_back(origin); + } + } ForEachBufferInRange(lower, upper - lower, [&](BufferId buffer_id, Buffer& buffer) { const DAddr buffer_addr = buffer.CpuAddr(); const DAddr begin = (std::max)(lower, buffer_addr); @@ -724,11 +749,21 @@ void BufferCache

::RecordReadDownload(std::span> r TakeEarlyLanded(own_addr, own_size, add_landed); } } - // EDEN_NX2_READ_CHECK: a check whose read's bytes other downloads bring is a download of its own. + // TEST: a check whose read's bytes other downloads bring is a download of its own. const bool check_only = !early && copies.empty() && check && !check_copies.empty(); if (copies.empty() && !check_only) { return; } + for (const Copy& entry : copies) { + GpuWaitStats::AddDownloadBytes(entry.copy.size); + } + if constexpr (HAS_READ_TRACE) { + if (ReadTrace::Enabled()) [[unlikely]] { + // Writes noted without a draw or dispatch after them end here at the latest. + TraceFlushWriteMarks(); + download.trace.start = runtime.TraceMark(); + } + } if (!copies.empty()) { auto staging = runtime.DownloadStagingBuffer(total_size_bytes, true); for (Copy& entry : copies) { @@ -746,7 +781,14 @@ void BufferCache

::RecordReadDownload(std::span> r } download.staging = staging; } - if (check && !check_copies.empty()) { + if constexpr (HAS_READ_TRACE) { + if (ReadTrace::Enabled()) [[unlikely]] { + download.trace.end = runtime.TraceMark(); + download.trace.sequence = ++trace_sequence; + download.trace.counts = runtime.TraceCounts(); + } + } + if (check && !check_copies.empty()) [[unlikely]] { // The same bytes again, from the same point of the GPU's work: what a synchronous // flush would have written. check->staging = runtime.DownloadStagingBuffer(check_size, true); @@ -776,6 +818,17 @@ void BufferCache

::RecordReadDownload(std::span> r } // A read waits for its copies; an early download's go with the next submission. download.tick = early ? runtime.QueueForHostRead() : runtime.SubmitForHostRead(); + if constexpr (HAS_READ_TRACE) { + if (ReadTrace::Enabled()) [[unlikely]] { + download.trace.tick = download.tick; + download.trace.flushed = ReadTrace::Now(); + download.trace.early = early; + if (early) { + ReadTrace::Count(ReadTrace::g_early.recorded); + ReadTrace::Count(ReadTrace::g_early.bytes, total_size_bytes); + } + } + } if (early) { ++pending_early_downloads; early_bytes_this_frame += total_size_bytes; @@ -786,50 +839,78 @@ void BufferCache

::RecordReadDownload(std::span> r } template -std::optional BufferCache

::FinishReadDownloads(DAddr device_addr, u64 size) { +void BufferCache

::FinishReadDownloads(DAddr device_addr, u64 size) { if constexpr (!IS_OPENGL) { std::unique_lock lock{mutex}; // The newest one over the range: those before it are written first. std::optional tick; + const ReadDownload* newest = nullptr; for (const ReadDownload& download : read_downloads) { - // EDEN_NX2_READ_CHECK: a check alone is waited for too, as its read's download. + // TEST (EDEN_NX2_READ_CHECK): a check alone is waited for too, as its read's download. if (download.write_ranges.Intersects(device_addr, size) || (download.check && download.check->own.Intersects(device_addr, size))) { tick = download.tick; + newest = &download; + } + } + // TEST (EDEN_NX2_READ_TRACE): the reading thread's read, when this is one. + ReadTrace::Read* trace = nullptr; + const auto trace_now = [&trace](u64 ReadTrace::Read::*field) { + if (trace != nullptr) [[unlikely]] { + if constexpr (HAS_READ_TRACE) { + trace->*field = ReadTrace::Now(); + } + } + }; + if constexpr (HAS_READ_TRACE) { + trace = ReadTrace::t_read; + if (trace != nullptr && newest != nullptr) [[unlikely]] { + TraceReadCopy(*trace, *newest, device_addr, size); } } if (!tick) { - return std::nullopt; + return; } - u64 waited = 0; if (!runtime.IsFree(*tick)) { + // TEST: a write from the GPU's side waits for an early download. + bool gpu_side = false; + u64 wait_start = 0; + if constexpr (HAS_READ_TRACE) { + gpu_side = ReadTrace::Enabled() && trace == nullptr && newest->early; + wait_start = gpu_side ? ReadTrace::Now() : 0; + } lock.unlock(); - const u64 wait_start = VideoCore::AsyncReadback::Now(); runtime.WaitForHostRead(*tick); - waited = (std::max)(VideoCore::AsyncReadback::Now() - wait_start, u64{1}); - VideoCore::AsyncReadback::CountHostWait(waited); + trace_now(&ReadTrace::Read::woke); + if constexpr (HAS_READ_TRACE) { + if (gpu_side) [[unlikely]] { + ReadTrace::Count(ReadTrace::g_early.gpu_side_waits); + ReadTrace::Count(ReadTrace::g_early.gpu_side_wait_ns, + ReadTrace::Now() - wait_start); + } + } lock.lock(); + } else { + trace_now(&ReadTrace::Read::woke); } - WriteReadDownloads(*tick, VideoCore::AsyncReadback::t_reading - ? VideoCore::AsyncReadback::Landing::Reader - : VideoCore::AsyncReadback::Landing::GpuSideWrite); - return waited; + trace_now(&ReadTrace::Read::locked); + WriteReadDownloads(*tick); } - return std::nullopt; } template void BufferCache

::OutdateReadDownloads(DAddr device_addr, u64 size) { - // EDEN_NX2_READ_CHECK: the GPU's copy gets newer bytes where an early download landed, + NoteReadCheckEvent(device_addr, size, ReadCheckEvent::Unrecorded); // TEST + // TEST (EDEN_NX2_READ_CHECK): the GPU's copy gets newer bytes where an early download landed, // which no read compares. A query result copy (SyncValues) is no GPU write the cache records: // a synchronous flush doesn't download it, and guest memory gets the result once the query // cache writes it there. MarkWrittenBuffer's bytes are GPU-modified from now on, which // TakeEarlyLanded leaves out, and every way out of that forgets them too. - if (!early_landed.empty()) { + if (!early_landed.empty()) [[unlikely]] { ForgetEarlyLanded(device_addr, size); } // The bytes stay GPU-modified when the downloads land, so a later read downloads them. - if (!read_downloads.empty()) { + if (!read_downloads.empty()) [[unlikely]] { for (ReadDownload& download : read_downloads) { download.clean_ranges.Subtract(device_addr, size); } @@ -859,6 +940,11 @@ bool BufferCache

::WriteFinishedReadDownloads(DAddr device_addr, u64 size) { template void BufferCache

::NoteWorkRecorded(bool dispatch) { if constexpr (!IS_OPENGL && USE_MEMORY_MAPS) { + if constexpr (HAS_READ_TRACE) { + if (ReadTrace::Enabled()) [[unlikely]] { + TraceWorkRecorded(dispatch); + } + } if (dispatch) { // Outside any render pass: the copies go right behind the dispatch. RecordEarlyDownloads(); @@ -875,7 +961,7 @@ void BufferCache

::NoteWorkRecorded(bool dispatch) { template void BufferCache

::NoteReadBack(DAddr begin, DAddr end, bool gpu_modified) { if constexpr (!IS_OPENGL && USE_MEMORY_MAPS) { - if (!gpu_modified && read_back_pages.empty()) { + if (!EarlyDownloadsEnabled() || (!gpu_modified && read_back_pages.empty())) { return; } for (DAddr page = begin; page < end; page += DEVICE_PAGESIZE) { @@ -897,6 +983,11 @@ void BufferCache

::NoteReadBack(DAddr begin, DAddr end, bool gpu_modified) { // Nothing new since its last read. continue; } + // TEST (EDEN_NX2_READ_TRACE): the bytes in guest memory before the read came. + if (it->landed_early && !gpu_modified && ReadTrace::Enabled()) [[unlikely]] { + ReadTrace::Count(ReadTrace::g_early.served_landed); + } + it->landed_early = false; it->written = false; it->unread_writes = 0; it->reads = (std::min)(static_cast(it->reads + 1), EarlyAfterReads); @@ -944,6 +1035,7 @@ void BufferCache

::NoteReadBackWrite(DAddr device_addr, u64 size) { &ReadBackPage::page); it != read_back_pages.end() && it->page < end; ++it) { it->written = true; + it->landed_early = false; if (it->unread_writes < MaxUnreadWrites) { ++it->unread_writes; } else { @@ -972,12 +1064,83 @@ void BufferCache

::RecordEarlyDownloads() { // Bounded: past it, the reads of these pages flush on demand. if (pending_early_downloads >= MaxPendingEarlyDownloads || early_bytes_this_frame >= EarlyBytesPerFrame) { + ++read_check.early_left_out; // TEST return; } RecordReadDownload(pages, pages, true); } } +template +void BufferCache

::TraceWorkRecorded(bool dispatch) { + if constexpr (HAS_READ_TRACE) { + runtime.TraceCountWork(dispatch); + TraceFlushWriteMarks(); + } +} + +template +void BufferCache

::TraceWriteRange(DAddr device_addr, u64 size) { + if constexpr (HAS_READ_TRACE) { + if (!cpu_read_ranges.Intersects(device_addr, size)) { + return; + } + trace_writes[trace_write_count++ % trace_writes.size()] = TraceWrite{ + .start = device_addr, + .end = device_addr + size, + .sequence = ++trace_sequence, + }; + trace_marks_pending = true; + } +} + +template +void BufferCache

::TraceFlushWriteMarks() { + if constexpr (HAS_READ_TRACE) { + if (!trace_marks_pending) { + return; + } + trace_marks_pending = false; + const ReadTrace::Write write{ + .tick = runtime.CurrentTick(), + .recorded = ReadTrace::Now(), + .mark = runtime.TraceMark(), + .counts = runtime.TraceCounts(), + }; + // The newest entries without one: those noted since the last mark. + const size_t count = (std::min)(trace_write_count, trace_writes.size()); + for (size_t i = 0; i < count; ++i) { + TraceWrite& entry = trace_writes[(trace_write_count - 1 - i) % trace_writes.size()]; + if (entry.write.recorded != 0) { + break; + } + entry.write = write; + } + } +} + +template +void BufferCache

::TraceReadCopy(ReadTrace::Read& read, const ReadDownload& download, + DAddr device_addr, u64 size) const { + if (download.trace.tick == 0) { + return; + } + read.has_copy = true; + read.copy = download.trace; + // The area's last GPU write before the copy. + const size_t count = (std::min)(trace_write_count, trace_writes.size()); + for (size_t i = 0; i < count; ++i) { + const TraceWrite& entry = trace_writes[(trace_write_count - 1 - i) % trace_writes.size()]; + if (entry.sequence > download.trace.sequence || entry.write.recorded == 0 || + entry.end <= device_addr || entry.start >= device_addr + size) { + continue; + } + read.has_write = true; + read.write = entry.write; + return; + } +} + template bool BufferCache

::EarlyDownloadServes(DAddr device_addr, u64 size) const { if (std::ranges::any_of(read_downloads, [&](const ReadDownload& download) { @@ -1003,6 +1166,7 @@ void BufferCache

::NoteEarlyLanded(const ReadDownload& download, if (early_landed.size() >= 4 * MaxReadBackPages) { early_landed.clear(); } + const u64 landed_at = read_check_event_count; download.clean_ranges.ForEach([&](DAddr start, DAddr end) { Common::RangeSet landed; landed.Add(start, end - start); @@ -1019,6 +1183,18 @@ void BufferCache

::NoteEarlyLanded(const ReadDownload& download, device_memory.ReadBlockUnsafe(from, entry.as_landed.data() + (from - page), to - from); entry.bytes.Add(from, to - from); + if (entry.from.size() >= 8) { + entry.from.erase(entry.from.begin()); + } + entry.from.push_back({ + .begin = from, + .end = to, + .tick = download.tick, + .copied = download.events_before, + .landed = landed_at, + .early = true, + .before_read = true, + }); } }); }); @@ -1041,7 +1217,7 @@ void BufferCache

::TakeEarlyLanded(DAddr device_addr, u64 size, Func&& func) { taken.Subtract(lower, upper - lower); }); taken.ForEach([&](DAddr lower, DAddr upper) { - func(lower, upper, it->second.as_landed.data() + (lower - page)); + func(lower, upper, it->second.as_landed.data() + (lower - page), it->second); it->second.bytes.Subtract(lower, upper - lower); }); if (it->second.bytes.Empty()) { @@ -1102,11 +1278,11 @@ void BufferCache

::ForgetEarlyLanded(DAddr device_addr, u64 size) { } template -void BufferCache

::WriteReadDownloads(u64 tick, VideoCore::AsyncReadback::Landing landed_by) { +void BufferCache

::WriteReadDownloads(u64 tick) { if constexpr (!IS_OPENGL) { while (!read_downloads.empty() && read_downloads.front().tick <= tick) { ReadDownload& download = read_downloads.front(); - // Guest memory this writes, and what its read check reads. The guest CPU writes + // Guest memory this writes, and TEST: what its check reads. The guest CPU writes // collected from now on store their bytes once this download is in guest memory. DAddr landing_begin = ~DAddr{0}; DAddr landing_end = 0; @@ -1116,7 +1292,7 @@ void BufferCache

::WriteReadDownloads(u64 tick, VideoCore::AsyncReadback::Land (std::max)(landing_end, static_cast(copy.src_offset + copy.size)); }; std::ranges::for_each(download.copies, cover); - if (download.check) { + if (download.check) [[unlikely]] { std::ranges::for_each(download.check->copies, cover); } const VideoCore::DownloadLanding landing{landing_begin, landing_end}; @@ -1134,23 +1310,33 @@ void BufferCache

::WriteReadDownloads(u64 tick, VideoCore::AsyncReadback::Land end - start); }); } - // EDEN_NX2_READ_CHECK: what early downloads brought, for the reads they serve. - if (download.early && ReadChecks()) { - NoteEarlyLanded(download, cpu_written); + // TEST: what early downloads brought, for the reads they serve. + if (download.early) { + if (ReadTrace::Enabled()) [[unlikely]] { + download.clean_ranges.ForEach([&](DAddr start, DAddr end) { + for (auto it = std::ranges::lower_bound( + read_back_pages, Common::AlignDown(start, DEVICE_PAGESIZE), {}, + &ReadBackPage::page); + it != read_back_pages.end() && it->page < end; ++it) { + it->landed_early = true; + } + }); + } + if (ReadChecks()) [[unlikely]] { + NoteEarlyLanded(download, cpu_written); + } } - // EDEN_NX2_READ_CHECK: what later reads' checks find of an earlier download where they kept its bytes. + // TEST: what later reads' checks find of an earlier download where they kept its bytes. for (size_t later = 1; later < read_downloads.size(); ++later) { - if (read_downloads[later].check) { - NoteLandedBytes(*read_downloads[later].check, download.write_ranges, - cpu_written); + if (read_downloads[later].check) [[unlikely]] { + NoteLandedBytes(*read_downloads[later].check, download, cpu_written); } } - if (download.check) { - CheckReadDownload(*download.check, cpu_written, landed_by); + if (download.check) [[unlikely]] { + CheckReadDownload(download, cpu_written); async_buffers_death_ring.push_back(download.check->staging); } - // A query result older than these bytes lands over them later: they stay - // GPU-modified, for a later read to download again. + // An older query result can still land over these bytes after this download. for (const PendingReport& report : pending_reports) { if (report.rewritten) { download.clean_ranges.Subtract(report.addr, report.size); @@ -1172,7 +1358,7 @@ void BufferCache

::WriteReadDownloads(u64 tick, VideoCore::AsyncReadback::Land } } }); - // Freed by the GPU thread (TickFrame). A read check alone has none. + // Freed by the GPU thread (TickFrame). TEST: a check alone has none. if (!download.copies.empty()) { async_buffers_death_ring.push_back(download.staging); } @@ -1209,9 +1395,9 @@ bool BufferCache

::ReadDownloadsCover(DAddr device_addr, u64 size) const { } template -void BufferCache

::CheckReadDownload(const ReadCheck& check, - const Common::RangeSet& cpu_written, - VideoCore::AsyncReadback::Landing landing) { +void BufferCache

::CheckReadDownload(const ReadDownload& download, + const Common::RangeSet& cpu_written) { + const ReadCheck& check = *download.check; // Guest memory now holds this download and the earlier ones: compare it with the synchronous // copy, except where a newer value replaced it since (CPU writes, ClearDownload). Writes // collected since the download was written count as newer too. Where an earlier download @@ -1232,8 +1418,8 @@ void BufferCache

::CheckReadDownload(const ReadCheck& check, Outdated, ///< An earlier download's bytes differ: the GPU wrote them since NotLanded, ///< No earlier download wrote it }; - const auto report = [this, &check, landing](Result result, DAddr address, u64 size, u8 found, - u8 wanted) { + const auto report = [this, &download](Result result, DAddr address, u64 size, u8 found, + u8 wanted) { if (result == Result::Rewritten) { read_check.rewritten += size; if (read_check.logged_rewrites++ < 16) { @@ -1246,7 +1432,6 @@ void BufferCache

::CheckReadDownload(const ReadCheck& check, return; } read_check.mismatched += size; - read_check.mismatched_by_landing[static_cast(landing)] += size; if (read_check.logged++ >= 16) { return; } @@ -1257,12 +1442,13 @@ void BufferCache

::CheckReadDownload(const ReadCheck& check, : (result == Result::NotLanded ? ": no earlier read's download wrote it" : ""); LOG_ERROR(HW_GPU, "Read download check: guest memory {:#x}+{:#x} differs from a synchronous " - "flush ({:#04x} for {:#04x} at its start){}; {} download of [{:#x}, {:#x}), " - "written to guest memory by {}", - address, size, found, wanted, why, - check.early ? "early" : (check.own.Empty() ? "read" : "check-only"), - check.read_begin, check.read_end, - VideoCore::AsyncReadback::landing_names[static_cast(landing)]); + "flush ({:#04x} for {:#04x} at its start){}", + address, size, found, wanted, why); + if (result == Result::Differs) { + ExplainReadCheckDifference(download, address, size, found, wanted); + } else if (result == Result::Outdated) { + ExplainReadCheckOutdated(download, address, size); + } }; std::vector guest; std::vector sources; @@ -1334,23 +1520,169 @@ void BufferCache

::CheckReadDownload(const ReadCheck& check, if (now >= read_check.last_log + 10) { read_check.last_log = now; LOG_INFO(HW_GPU, - "Read download check (EDEN_NX2_READ_CHECK): {} downloads compared " - "with a synchronous flush, {} KiB; {} bytes differ (landed by the reading " - "thread {}, by a GPU-side write {}, after the host GPU ran them {}), {} " - "GPU-modified bytes no download brought (both must stay 0); {} bytes got newer " - "values after an earlier read's download brought them (not compared)", + "Read download check (EDEN_NX2_READ_CHECK): {} downloads compared with a " + "synchronous flush, {} KiB; {} bytes differ, {} GPU-modified bytes no download " + "brought (both must stay 0); {} bytes got newer values after an earlier read's " + "download brought them (not compared); {} guest CPU writes waited for a " + "download landing under them, {:.3f} ms; recorded: {} read downloads, {} check " + "copies alone, {} early downloads ({} left out at their bounds); {} reads an " + "early download served flushed for the check", read_check.checked, read_check.compared / 1024, read_check.mismatched, - read_check.mismatched_by_landing[0], read_check.mismatched_by_landing[1], - read_check.mismatched_by_landing[2], read_check.uncovered, read_check.rewritten); + read_check.uncovered, read_check.rewritten, VideoCore::DownloadLanding::Waits(), + static_cast(VideoCore::DownloadLanding::WaitNs()) / 1e6, + read_check.reads_recorded, read_check.alone_recorded, read_check.early_recorded, + read_check.early_left_out, read_check.early_served); } } template -void BufferCache

::NoteLandedBytes(ReadCheck& check, const Common::RangeSet& written, +void BufferCache

::ExplainReadCheckDifference(const ReadDownload& download, DAddr address, + u64 size, u8 found, u8 wanted) { + if constexpr (!IS_OPENGL) { + // What the download's own copy holds there: the check's copy was recorded right after it, + // so they differ only if the GPU wrote between the two. Equal, guest memory changed after + // the download wrote it. + std::optional brought; + for (const BufferCopy& copy : download.copies) { + if (address >= copy.src_offset && address < copy.src_offset + copy.size) { + brought = download.staging.mapped_span[copy.dst_offset - download.staging.offset + + (address - copy.src_offset)]; + } + } + std::string source = "the download has no copy there"; + if (brought) { + source = fmt::format( + "the download brought {:#04x}{}", *brought, + *brought == wanted ? ", as the check copy: guest memory changed after it landed" + : *brought == found + ? ", as guest memory holds: the GPU wrote between the download's copy and the " + "check's" + : ", neither"); + } + DAddr buffer_addr = 0; + u64 buffer_size = 0; + ForEachBufferInRange(address, 1, [&](BufferId, Buffer& buffer) { + buffer_addr = buffer.CpuAddr(); + buffer_size = buffer.SizeBytes(); + }); + const bool collected = PendingCpuWriteRanges().Intersects(address, size); + LOG_ERROR(HW_GPU, + "Read download check: {:#x}+{:#x} is in {} download of tick {} (now {}), buffer " + "{:#x}+{:#x} at offset {:#x}: {}; {}; newest GPU-side guest memory write noted " + "there: {}", + address, size, download.early ? "an early" : "a read's", download.tick, + runtime.CurrentTick(), buffer_addr, buffer_size, address - buffer_addr, source, + collected ? "a guest CPU write is collected there now" + : "no guest CPU write is collected there", + VideoCore::ReadCheckWrites::Find(address, size)); + } +} + +template +void BufferCache

::ExplainReadCheckOutdated(const ReadDownload& download, DAddr address, + u64 size) { + if constexpr (!IS_OPENGL) { + // The newest download noted as bringing it: early_landed before this read came (a), or + // one on its way at this read that landed before it (b). + const typename ReadCheck::Origin* origin = nullptr; + for (const auto& entry : download.check->origins) { + if (address >= entry.begin && address < entry.end) { + origin = &entry; + } + } + std::string path = "no download noted as bringing it"; + u64 since = download.events_before; + u64 landed_at = download.events_before; + if (origin != nullptr) { + path = fmt::format("{} {} download of tick {}", + origin->before_read ? "(a) landed before this read came, by" + : "(b) on its way at this read, landed by", + origin->early ? "an early" : "a read's", origin->tick); + since = origin->copied; + landed_at = origin->landed; + } + // What wrote it since that download's copy, oldest first. + static constexpr std::array kinds{ + "GPU write", "unrecorded GPU write (query result copy)", + "DMA clear", "guest write", + "upload", "inline upload", + }; + const u64 total = read_check_event_count; + const u64 ring = read_check_events.size(); + // No ring yet: no event was noted (NoteReadCheckEvent sizes it first). + const u64 first = + ring == 0 ? total : (std::max)(since, total > ring ? total - ring : u64{0}); + std::string events; + u64 listed = 0; + u64 uploads = 0; + bool guest_write_since = false; + bool upload_since = false; + for (u64 index = first; index < total; ++index) { + const ReadCheckEventEntry& event = read_check_events[index % ring]; + if (event.begin >= address + size || address >= event.end) { + continue; + } + const bool after_landing = event.sequence > landed_at; + const bool upload = + event.kind == ReadCheckEvent::Upload || event.kind == ReadCheckEvent::Inline; + uploads += upload ? 1 : 0; + upload_since = upload_since || (upload && after_landing); + guest_write_since = + guest_write_since || (event.kind == ReadCheckEvent::GuestWrite && after_landing); + if (listed++ >= 8) { + continue; + } + events += fmt::format( + "{}{}{} ({} it landed, {} this read's copy)", events.empty() ? "" : ", ", + kinds[static_cast(event.kind)], + event.kind == ReadCheckEvent::GpuWrite + ? fmt::format(" by {}", + GpuWaitStats::write_site_names[static_cast(event.site)]) + : std::string{}, + after_landing ? "after" : "before", + event.sequence > download.events_before ? "after" : "before"); + } + if (listed > 8) { + events += fmt::format(" and {} more", listed - 8); + } + const DAddr page = Common::AlignDown(address, DEVICE_PAGESIZE); + LOG_ERROR(HW_GPU, + "Read download check: {:#x}+{:#x} is outdated: {}; compared by {} download of " + "tick {} (now {}); since that copy over it: {}{}; a guest write since it " + "landed: {}; uploads since that copy: {}, since it landed: {}; newest GPU-side " + "guest memory write noted there: {}; page {:#x} in the tracker: CPU-modified " + "{}, GPU-modified {}; GPU-modified in the cache: {}; a guest CPU write " + "collected there now: {}", + address, size, path, download.early ? "an early" : "a read's", download.tick, + runtime.CurrentTick(), events.empty() ? "nothing noted" : events, + since < first ? " (older events lost)" : "", guest_write_since ? "yes" : "no", + uploads, upload_since ? "yes" : "no", + VideoCore::ReadCheckWrites::Find(address, size), page, + memory_tracker.IsRegionCpuModified(page, DEVICE_PAGESIZE) ? "yes" : "no", + memory_tracker.IsRegionGpuModified(page, DEVICE_PAGESIZE) ? "yes" : "no", + gpu_modified_ranges.Intersects(address, size) ? "yes" : "no", + PendingCpuWriteRanges().Intersects(address, size) ? "yes" : "no"); + } +} + +template +void BufferCache

::NoteLandedBytes(ReadCheck& check, const ReadDownload& download, const Common::RangeSet& cpu_written) { if (check.kept.Empty()) { return; } + const Common::RangeSet& written = download.write_ranges; + const auto note_origin = [&](DAddr start, DAddr end) { + check.origins.push_back({ + .begin = start, + .end = end, + .tick = download.tick, + .copied = download.events_before, + .landed = read_check_event_count, + .early = download.early, + .before_read = false, + }); + }; for (const BufferCopy& copy : check.copies) { const DAddr copy_addr = static_cast(copy.src_offset); u8* const landed = check.landed_bytes.data() + (copy.dst_offset - check.staging.offset); @@ -1358,6 +1690,7 @@ void BufferCache

::NoteLandedBytes(ReadCheck& check, const Common::RangeSet::NoteLandedBytes(ReadCheck& check, const Common::RangeSet void BufferCache

::ClearDownload(DAddr device_addr, u64 size) { - if (const std::optional waited = FinishReadDownloads(device_addr, size)) { - VideoCore::AsyncReadback::CountClearLanding(*waited); - } + FinishReadDownloads(device_addr, size); ClearReplacedDownload(device_addr, size); } template void BufferCache

::ClearReplacedDownload(DAddr device_addr, u64 size) { ClearFenceDownloads(device_addr, size); - // EDEN_NX2_READ_CHECK: a newer value there, as without early downloads. - if (!early_landed.empty()) { + // TEST (EDEN_NX2_READ_CHECK): a newer value there, as without early downloads. + if (!early_landed.empty()) [[unlikely]] { ForgetEarlyLanded(device_addr, size); } - if (!read_downloads.empty()) { + if (!read_downloads.empty()) [[unlikely]] { for (ReadDownload& download : read_downloads) { download.write_ranges.Subtract(device_addr, size); download.clean_ranges.Subtract(device_addr, size); - if (download.check) { + if (download.check) [[unlikely]] { download.check->changed.Add(device_addr, size); } } @@ -1422,6 +1753,8 @@ bool BufferCache

::DMACopy(GPUVAddr src_address, GPUVAddr dest_address, u64 am ClearDownload(*cpu_dest_address, amount); MarkVertexBuffersStale(*cpu_dest_address, amount); + NoteReadCheckEvent(*cpu_dest_address, amount, ReadCheckEvent::GpuWrite, + GpuWaitStats::WriteSite::DmaCopy); // TEST BufferId buffer_a; BufferId buffer_b; @@ -1440,6 +1773,11 @@ bool BufferCache

::DMACopy(GPUVAddr src_address, GPUVAddr dest_address, u64 am .size = amount, }}; + // Guest memory gets the copy below, which holds every byte the source has in guest memory: + // command list pages in the destination read those from it, and their copies are stale. The + // bytes copied from GPU-written source ranges are noted as GPU writes after this. + Nx2NoteListHostWrite(*cpu_dest_address, amount); + boost::container::small_vector, 4> tmp_intervals; auto mirror = [&](DAddr base_address, DAddr base_address_end) { const u64 size = base_address_end - base_address; @@ -1454,7 +1792,11 @@ bool BufferCache

::DMACopy(GPUVAddr src_address, GPUVAddr dest_address, u64 am const bool has_new_downloads = tmp_intervals.size() != 0; for (const auto& pair : tmp_intervals) { gpu_modified_ranges.Add(pair.first, pair.second); + GpuWaitStats::NoteGpuWrite(pair.first, pair.second, GpuWaitStats::WriteSite::DmaCopy); NoteReadBackWrite(pair.first, pair.second); + if (ReadTrace::Enabled()) [[unlikely]] { + TraceWriteRange(pair.first, pair.second); + } if (Tegra::g_nx2_ampere.load(std::memory_order_relaxed)) { Tegra::Nx2MarkGpuWritten(pair.first, pair.second); Nx2NoteListWrite(buffer_b, pair.first, pair.second); @@ -1464,6 +1806,9 @@ bool BufferCache

::DMACopy(GPUVAddr src_address, GPUVAddr dest_address, u64 am src_buffer.MarkUsage(copy.src_offset, copy.size); dest_buffer.MarkUsage(copy.dst_offset, copy.size); runtime.CopyBuffer(dest_buffer, src_buffer, copies, true); + if (ReadTrace::Enabled()) [[unlikely]] { + TraceFlushWriteMarks(); + } if (has_new_downloads) { memory_tracker.MarkRegionAsGpuModified(*cpu_dest_address, amount); NoteUniformWrite(*cpu_dest_address, amount); @@ -1485,14 +1830,21 @@ bool BufferCache

::DMAClear(GPUVAddr dst_address, u64 amount, u32 value) { if (!cpu_dst_address) { return false; } - const bool dest_dirty = IsRegionRegistered(*cpu_dst_address, amount); + // `amount` counts words. + const size_t size = amount * sizeof(u32); + const bool dest_dirty = IsRegionRegistered(*cpu_dst_address, size); if (!dest_dirty) { return false; } - const size_t size = amount * sizeof(u32); + // The caller writes the cleared words to guest memory after this: earlier reads' downloads + // land first, so a reader of these bytes never finds neither there. + FinishReadDownloads(*cpu_dst_address, size); ClearDownload(*cpu_dst_address, size); gpu_modified_ranges.Subtract(*cpu_dst_address, size); + NoteReadCheckEvent(*cpu_dst_address, size, ReadCheckEvent::DmaClear); // TEST + // As an inline write: command list pages read the cleared words from guest memory. + Nx2NoteListHostWrite(*cpu_dst_address, size); const BufferId buffer = FindBuffer(*cpu_dst_address, static_cast(size), false); Buffer& dest_buffer = slot_buffers[buffer]; @@ -1502,6 +1854,22 @@ bool BufferCache

::DMAClear(GPUVAddr dst_address, u64 amount, u32 value) { return true; } +template +void BufferCache

::MarkUsed(DAddr device_addr, u64 size) { + const u64 page = device_addr >> CACHING_PAGEBITS; + if (device_addr == 0 || size == 0 || page >= page_table.size()) { + return; + } + const BufferId buffer_id = page_table[page]; + if (!buffer_id) { + return; + } + Buffer& buffer = slot_buffers[buffer_id]; + if (buffer.IsInBounds(device_addr, size)) { + buffer.MarkUsage(buffer.Offset(device_addr), size); + } +} + template std::pair BufferCache

::ObtainBuffer(GPUVAddr gpu_addr, u32 size, ObtainBufferSynchronize sync_info, @@ -1528,10 +1896,14 @@ std::pair BufferCache

::ObtainCPUBuffer( break; } + // Each caller records GPU work that reads or writes these bytes now (indirect dispatches, DMA + // copies between images and buffers, query results, conditional rendering): an upload of them + // later in this submission must run after it. + buffer.MarkUsage(buffer.Offset(device_addr), size); switch (post_op) { case ObtainBufferOperation::MarkAsWritten: - MarkWrittenBuffer(buffer_id, device_addr, size); - buffer.MarkUsage(buffer.Offset(device_addr), size); + MarkWrittenBuffer(buffer_id, device_addr, size, false, + GpuWaitStats::WriteSite::ImageToBuffer); break; case ObtainBufferOperation::DiscardWrite: { ClearDownload(device_addr, size); @@ -1641,7 +2013,13 @@ void BufferCache

::BindHostGeometryBuffers(bool is_indexed) { template void BufferCache

::BindHostStageBuffers(size_t stage) { - BindHostGraphicsUniformBuffers(stage); + { + const Nx2Profile::FineScope uniform_scope{Nx2Profile::Zone::UniformBuffers}; + BindHostGraphicsUniformBuffers(stage); + } + Nx2Profile::AddFine( + Nx2Profile::FineCounter::StorageBinds, + static_cast(std::popcount(channel_state->enabled_storage_buffers[stage]))); if (channel_state->enabled_storage_buffers[stage] != 0) { BindHostGraphicsStorageBuffers(stage); } @@ -1909,7 +2287,10 @@ void BufferCache

::BindComputeStorageBuffer(size_t ssbo_index, u32 cbuf_index, const std::array block{qmd.block_dim_x.Value(), qmd.block_dim_y.Value(), qmd.block_dim_z.Value()}; std::optional> grid; - if (!kepler_compute->GetIndirectComputeAddress()) { + // A dispatch from GPU-written list words (Nx2ListIndirect) has the slot's placeholders + // in its grid, not the counts the host GPU dispatches: its grid is unknown here too. + if (!kepler_compute->GetIndirectComputeAddress() && + !kepler_compute->GetIndirectListSlot()) { grid = std::array{qmd.grid_dim_x.Value(), qmd.grid_dim_y.Value(), qmd.grid_dim_z.Value()}; } @@ -2115,8 +2496,8 @@ Common::RangeSet BufferCache

::PendingCpuWriteRanges() { template void BufferCache

::WriteDownloadedMemory(const Common::RangeSet& cpu_written, DAddr device_addr, const u8* data, u64 size) { - // EDEN_NX2_READ_CHECK: newer GPU bytes than an early download brought before. - if (!early_landed.empty()) { + // TEST (EDEN_NX2_READ_CHECK): newer GPU bytes than an early download brought before. + if (!early_landed.empty()) [[unlikely]] { ForgetEarlyLanded(device_addr, size); } if (!cpu_written.Intersects(device_addr, size)) { @@ -2431,6 +2812,10 @@ void BufferCache

::BindHostDrawIndirectBuffers() { Buffer& buffer = slot_buffers[binding.buffer_id]; TouchBuffer(buffer, binding.buffer_id); SynchronizeBuffer(buffer, binding.device_addr, binding.size); + // The draw reads the arguments (GetDrawIndirectBuffer, GetDrawIndirectCount). + if (binding.buffer_id != NULL_BUFFER_ID) { + buffer.MarkUsage(buffer.Offset(binding.device_addr), binding.size); + } }; if (current_draw_indirect->include_count) { bind_buffer(channel_state->count_buffer_binding); @@ -2475,6 +2860,8 @@ void BufferCache

::BindHostGraphicsUniformBuffer(size_t stage, u32 index, u32 } channel_state->fast_bound_uniform_buffers[stage] |= 1u << binding_index; channel_state->uniform_buffer_binding_sizes[stage][binding_index] = size; + Nx2Profile::AddFine(Nx2Profile::FineCounter::UniformOther, 1); + Nx2Profile::AddFine(Nx2Profile::FineCounter::UniformBytes, size); return; } const Binding& binding = channel_state->uniform_buffers[stage][index]; @@ -2502,6 +2889,8 @@ void BufferCache

::BindHostGraphicsUniformBuffer(size_t stage, u32 index, u32 channel_state->uniform_buffer_binding_sizes[stage][binding_index] = size; uniform_keep.streamed[stage] |= 1u << index; ++uniform_keep.kept; + Nx2Profile::AddFine(Nx2Profile::FineCounter::UniformRebinds, 1); + Nx2Profile::AddFine(Nx2Profile::FineCounter::UniformBytes, size); return; } } @@ -2522,7 +2911,12 @@ void BufferCache

::BindHostGraphicsUniformBuffer(size_t stage, u32 index, u32 // Native programs keep tables next to their code in a 64 KiB bank read with indirect offsets // (Splatoon Raiders: the dialogue box corners in c1). Writes to that memory can go unnoticed by // the memory tracker, leaving stale host contents, so upload such a bank again once per frame. - if (has_host_buffer && (*channel_state->uniform_buffer_sizes)[stage][index] == 0x10000 && + // Ocarina of Time is left out (unless EDEN_NX2_PERF_VERIFY=1): a compare of its banks against + // shadow copies found no page changed behind the tracker, and the uploads cost frame rate. + constexpr u64 ocarina_of_time = 0x0400486024DEC000ULL; + const bool bank_recheck = program_id != ocarina_of_time || Nx2Perf::Verify(); + if (bank_recheck && has_host_buffer && + (*channel_state->uniform_buffer_sizes)[stage][index] == 0x10000 && size != 0 && gpu_memory != nullptr && gpu_memory->UsesNativeTiling() && (Nx2Perf::Verify() || !native_indirect_uniforms_this_frame.contains(device_addr)) && !memory_tracker.IsRegionGpuModified(device_addr, size) && @@ -2546,6 +2940,7 @@ void BufferCache

::BindHostGraphicsUniformBuffer(size_t stage, u32 index, u32 const bool use_fast_buffer = needs_alignment_stream || (has_host_buffer && size <= channel_state->uniform_buffer_skip_cache_size && !memory_tracker.IsRegionGpuModified(device_addr, size)); + Nx2Profile::AddFine(Nx2Profile::FineCounter::UniformBytes, size); if (use_fast_buffer) { if constexpr (IS_OPENGL) { if (runtime.HasFastBufferSubData()) { @@ -2561,6 +2956,7 @@ void BufferCache

::BindHostGraphicsUniformBuffer(size_t stage, u32 index, u32 } const auto span = ImmediateBufferWithData(device_addr, size); runtime.PushFastUniformBuffer(stage, binding_index, span); + Nx2Profile::AddFine(Nx2Profile::FineCounter::UniformOther, 1); return; } } @@ -2573,8 +2969,19 @@ void BufferCache

::BindHostGraphicsUniformBuffer(size_t stage, u32 index, u32 // stream allocation again instead of allocating and copying (uniform_stream_reuse.h). // EDEN_NX2_UBOREUSE=0 turns it off. if (VideoCommon::UniformStreamReuseEnabled()) { + // TEST: rebinds counted (EDEN_NX2_GPU_PROFILE=2). + const u64 rebinds = + VideoCommon::UniformStreamReuseState() + .rebinds; streamed = VideoCommon::BindSmallUniformReusing( runtime, device_memory, stage, index, binding_index, device_addr, size); + const u64 rebound = + VideoCommon::UniformStreamReuseState() + .rebinds - + rebinds; + Nx2Profile::AddFine(Nx2Profile::FineCounter::UniformRebinds, rebound); + Nx2Profile::AddFine(Nx2Profile::FineCounter::UniformStreamed, + streamed ? 1 - rebound : 0); if (streamed) { uniform_keep.streamed[stage] |= 1u << index; ++uniform_keep.streamed_binds; @@ -2584,9 +2991,11 @@ void BufferCache

::BindHostGraphicsUniformBuffer(size_t stage, u32 index, u32 if (!streamed) { const std::span span = runtime.BindMappedUniformBuffer(stage, binding_index, size); device_memory.ReadBlockUnsafe(device_addr, span.data(), size); + Nx2Profile::AddFine(Nx2Profile::FineCounter::UniformStreamed, 1); } return; } + Nx2Profile::AddFine(Nx2Profile::FineCounter::UniformCached, 1); // Classic cached path if (verify_kept && uniform_keep.mismatches++ < 16) { LOG_ERROR(HW_GPU, @@ -2701,7 +3110,8 @@ bool BufferCache

::BindMultiRangeStorage(const Binding& binding, bool is_writt // write_mark_limit: the native write mark (BindComputeStorageBuffer), spread over // the segments in order. const u32 mark = static_cast(std::min(segment.size, write_mark_limit)); - MarkWrittenBuffer(segment.buffer_id, segment.device_addr, mark); + MarkWrittenBuffer(segment.buffer_id, segment.device_addr, mark, false, + GpuWaitStats::WriteSite::MultiRangeStorage); if (write_mark_limit != ~u64{0}) { write_mark_limit -= mark; } @@ -2732,7 +3142,8 @@ void BufferCache

::BindHostGraphicsStorageBuffers(size_t stage) { buffer.MarkUsage(offset, size); if (is_written) { - MarkWrittenBuffer(binding.buffer_id, binding.device_addr, size); + MarkWrittenBuffer(binding.buffer_id, binding.device_addr, size, false, + GpuWaitStats::WriteSite::GraphicsStorage); } if constexpr (NEEDS_BIND_STORAGE_INDEX) { @@ -2754,7 +3165,8 @@ void BufferCache

::BindHostGraphicsTextureBuffers(size_t stage) { const bool is_written = ((channel_state->written_texture_buffers[stage] >> index) & 1) != 0; if (is_written) { - MarkWrittenBuffer(binding.buffer_id, binding.device_addr, size); + MarkWrittenBuffer(binding.buffer_id, binding.device_addr, size, false, + GpuWaitStats::WriteSite::GraphicsTexel); } const u32 offset = buffer.Offset(binding.device_addr); @@ -2792,7 +3204,8 @@ void BufferCache

::BindHostTransformFeedbackBuffers() { TouchBuffer(buffer, binding.buffer_id); size = binding.size; SynchronizeBuffer(buffer, binding.device_addr, size); - MarkWrittenBuffer(binding.buffer_id, binding.device_addr, size); + MarkWrittenBuffer(binding.buffer_id, binding.device_addr, size, false, + GpuWaitStats::WriteSite::TransformFeedback); offset = buffer.Offset(binding.device_addr); buffer.MarkUsage(offset, size); host_buffer = &buffer; @@ -2839,11 +3252,16 @@ void BufferCache

::BindHostComputeUniformBuffers() { needs_alignment_stream || (has_host_buffer && size <= channel_state->uniform_buffer_skip_cache_size && !memory_tracker.IsRegionGpuModified(binding.device_addr, size)); + // GPU-written list words owed there go over their placeholders (Nx2ListIndirect). + const auto guest_word = [this](DAddr word) { return device_memory.Read(word); }; if constexpr (!IS_OPENGL) { if (use_fast_buffer) { const std::span span = runtime.BindMappedUniformBuffer(0, binding_index, size); device_memory.ReadBlockUnsafe(binding.device_addr, span.data(), size); + const auto stream = runtime.LastStreamUniformRef(); + runtime.Nx2PatchComputeUniform(binding.device_addr, size, stream.buffer, + stream.offset, guest_word); return; } } @@ -2851,6 +3269,12 @@ void BufferCache

::BindHostComputeUniformBuffers() { SynchronizeBuffer(buffer, binding.device_addr, size); buffer.MarkUsage(offset, size); + if constexpr (!IS_OPENGL) { + runtime.Nx2PatchComputeUniform(binding.device_addr, size, buffer.Handle(), offset, + guest_word); + } else { + (void)guest_word; + } if constexpr (NEEDS_BIND_UNIFORM_INDEX) { runtime.BindComputeUniformBuffer(binding_index, buffer, offset, size); ++binding_index; @@ -2889,7 +3313,8 @@ void BufferCache

::BindHostComputeStorageBuffers() { const u32 mark = is_written ? static_cast(std::min(size, write_mark_limit)) : 0; if (mark != 0) { - MarkWrittenBuffer(binding.buffer_id, binding.device_addr, mark, true); + MarkWrittenBuffer(binding.buffer_id, binding.device_addr, mark, true, + GpuWaitStats::WriteSite::ComputeStorage); } if constexpr (NEEDS_BIND_STORAGE_INDEX) { @@ -2912,7 +3337,8 @@ void BufferCache

::BindHostComputeTextureBuffers() { const bool is_written = ((channel_state->written_compute_texture_buffers >> index) & 1) != 0; if (is_written) { - MarkWrittenBuffer(binding.buffer_id, binding.device_addr, size); + MarkWrittenBuffer(binding.buffer_id, binding.device_addr, size, false, + GpuWaitStats::WriteSite::ComputeTexel); } const u32 offset = buffer.Offset(binding.device_addr); @@ -2994,18 +3420,62 @@ void BufferCache

::UpdateIndexBuffer() { const GPUVAddr gpu_addr_begin = index_buffer_ref.StartAddress(); const GPUVAddr gpu_addr_end = index_buffer_ref.EndAddress(); - const std::optional device_addr = gpu_memory->GpuToCpuAddress(gpu_addr_begin); - const u32 address_size = static_cast(gpu_addr_end - gpu_addr_begin); const u32 draw_size = (index_buffer_ref.count + index_buffer_ref.first) * u32(index_buffer_ref.FormatSizeInBytes()); + // The draw macros mark the index buffer dirty on every draw, mostly with the same buffer and + // another count: the address and the buffer found are kept while the range, the GPU mappings + // and the buffers are, for any size up to the one the buffer was found for. + IndexBufferMemo& memo = index_buffer_memo; + const u64 map_serial = gpu_memory->MapSerial(); + const u32 address_size = static_cast(gpu_addr_end - gpu_addr_begin); const u32 size = (std::min)(address_size, draw_size); + const bool memo_hit = memo.channel == channel_state && memo.memory == gpu_memory && + memo.begin == gpu_addr_begin && memo.end == gpu_addr_end && + memo.map_serial == map_serial && memo.deletions == buffer_deletions && + size != 0 && size <= memo.found_size && + !Settings::IsGPUFenceBehaviorAccurate(); // FindBuffer waits for writes + if (memo_hit && !Nx2Perf::Verify()) { + channel_state->index_buffer = Binding{ + .device_addr = memo.device_addr, + .size = size, + .buffer_id = memo.buffer_id, + }; + return; + } + const std::optional device_addr = gpu_memory->GpuToCpuAddress(gpu_addr_begin); if (size == 0 || !device_addr) { channel_state->index_buffer = NULL_BINDING; return; } + const BufferId buffer_id = FindBuffer(*device_addr, size, false); channel_state->index_buffer = Binding{ .device_addr = *device_addr, .size = size, - .buffer_id = FindBuffer(*device_addr, size, false), + .buffer_id = buffer_id, + }; + if (memo_hit) { + if (memo.device_addr != *device_addr || memo.buffer_id != buffer_id) { + static std::atomic reports{}; + if (reports.fetch_add(1, std::memory_order_relaxed) < 16) { + LOG_ERROR(HW_GPU, + "Nx2Perf index buffer memo mismatch: {:#x}+{:#x} kept {:#x} buffer {}, " + "found {:#x} buffer {}", + gpu_addr_begin, size, memo.device_addr, memo.buffer_id.index, + *device_addr, buffer_id.index); + } + } + return; + } + // FindBuffer may have deleted buffers (joins): the deletion count is read after it. + memo = IndexBufferMemo{ + .channel = channel_state, + .memory = gpu_memory, + .begin = gpu_addr_begin, + .end = gpu_addr_end, + .map_serial = map_serial, + .deletions = buffer_deletions, + .device_addr = *device_addr, + .found_size = size, + .buffer_id = buffer_id, }; } @@ -3192,7 +3662,12 @@ void BufferCache

::UpdateComputeTextureBuffers() { } template -void BufferCache

::MarkWrittenBuffer(BufferId buffer_id, DAddr device_addr, u32 size, bool needs_sync) { +void BufferCache

::MarkWrittenBuffer(BufferId buffer_id, DAddr device_addr, u32 size, + bool needs_sync, GpuWaitStats::WriteSite site) { + GpuWaitStats::NoteGpuWrite(device_addr, size, site); + if (ReadTrace::Enabled()) [[unlikely]] { + TraceWriteRange(device_addr, size); + } if constexpr (!IS_OPENGL) { if (needs_sync) { Buffer& buffer = slot_buffers[buffer_id]; @@ -3205,6 +3680,7 @@ void BufferCache

::MarkWrittenBuffer(BufferId buffer_id, DAddr device_addr, u3 for (auto& ranges : pending_download_ranges) { ranges.Subtract(device_addr, size); } + NoteReadCheckEvent(device_addr, size, ReadCheckEvent::GpuWrite, site); // TEST OutdateReadDownloads(device_addr, size); NoteReadBackWrite(device_addr, size); memory_tracker.MarkRegionAsGpuModified(device_addr, size); @@ -3242,6 +3718,11 @@ void BufferCache

::Nx2NoteListWrite(BufferId buffer_id, DAddr device_addr, u64 std::scoped_lock lock{nx2_list_mutex}; Nx2ListPage& entry = nx2_list_pages[page_addr]; ++entry.writes; + Nx2MarkListWords( + entry, (std::max)(device_addr, page_addr) - page_addr, + (std::min)(device_addr + size, page_addr + (DAddr{1} << page_bits)) - + page_addr, + true); if (!nx2_collect_list_copies && !entry.draw_written) { entry.draw_written = true; LOG_INFO(HW_GPU, "Command list page {:#x} is written by draws too: its " @@ -3256,6 +3737,9 @@ void BufferCache

::Nx2NoteListWrite(BufferId buffer_id, DAddr device_addr, u64 Tegra::Nx2CmdPageDiag::NoteWrite(page_addr, device_addr, size, !nx2_collect_list_copies); } + if (Tegra::Nx2ListStats::Any()) { + Tegra::Nx2ListStats::NoteWrite(page_addr, device_addr, size); + } } ++page; } @@ -3295,6 +3779,9 @@ bool BufferCache

::Nx2CopyListPages(u64 tick) { buffer.MarkUsage(copy.src_offset, copy.size); runtime.CopyBuffer(staging.buffer, buffer, std::span{©, 1}, true); recorded = true; + if (Tegra::Nx2ListStats::Any()) { + Tegra::Nx2ListStats::NoteCopy(page); + } std::scoped_lock lock{nx2_list_mutex}; Nx2ListPage& entry = nx2_list_pages[page]; if (entry.copy) { @@ -3355,8 +3842,15 @@ auto BufferCache

::Nx2ReadListCopy(DAddr device_addr, std::span out, bool return entry.synced_writes == entry.writes ? Nx2ListRead::Synced : Nx2ListRead::Unknown; } - // Pages the CPU writes the lists of (GPU-written once, long before) stay synced. - if (entry.synced_writes == entry.writes) { + // Pages the CPU writes the lists of (GPU-written once, long before) stay synced, and + // a list on none of the words GPU work wrote since the page was flushed is in guest + // memory too: it waits for no copy. + if (entry.synced_writes == entry.writes || + (!Tegra::Nx2ListStats::t_reference && entry.flushed && + !Nx2HasListWords(entry, offset, offset + out.size()))) { + if (entry.synced_writes != entry.writes) { + Tegra::Nx2ListStats::NoteServed(Tegra::Nx2ListStats::Served::WordsClean); + } return Nx2ListRead::Synced; } if (!current(entry)) { @@ -3364,7 +3858,7 @@ auto BufferCache

::Nx2ReadListCopy(DAddr device_addr, std::span out, bool } // The lists after the first of a batch find the copy done: read it at once. if (runtime.IsFree(entry.copy_tick)) { - std::memcpy(out.data(), entry.copy->mapped_span.data() + offset, out.size()); + Nx2ReadListWords(entry, device_addr, offset, out); return Nx2ListRead::Copied; } tick = entry.copy_tick; @@ -3375,24 +3869,83 @@ auto BufferCache

::Nx2ReadListCopy(DAddr device_addr, std::span out, bool } } // Submitted with the work that wrote the page (Nx2CopyListPages's caller flushes). + const u64 wait_start = Tegra::Nx2ListStats::Any() ? Tegra::Nx2ListStats::NowNs() : 0; runtime.WaitSubmittedTick(tick); + if (Tegra::Nx2ListStats::Any()) { + Tegra::Nx2ListStats::NoteServed(Tegra::Nx2ListStats::Served::CopyWaited, wait_start, + Tegra::Nx2ListStats::NowNs() - wait_start); + } std::scoped_lock lock{nx2_list_mutex}; const auto it = nx2_list_pages.find(page); if (it == nx2_list_pages.end() || !current(it->second) || it->second.copy_tick != tick || it->second.copy_writes != writes) { return Nx2ListRead::Unknown; } - std::memcpy(out.data(), it->second.copy->mapped_span.data() + offset, out.size()); + Nx2ReadListWords(it->second, device_addr, offset, out); return Nx2ListRead::Copied; } } +template +void BufferCache

::Nx2ReadListWords(const Nx2ListPage& entry, DAddr device_addr, u64 offset, + std::span out) { + if constexpr (!IS_OPENGL) { + // The copy holds the words GPU work wrote. Guest memory holds the others once the page + // was flushed: the host wrote them, and the buffer the copy came from need not have had + // those writes uploaded. EDEN_NX2_LIST_VERIFY's read as before takes the whole copy. + const u8* const copy = entry.copy->mapped_span.data(); + const u64 end = offset + out.size(); + bool all_gpu = true; + for (u64 word = offset / 4; word < (end + 3) / 4 && all_gpu; ++word) { + all_gpu = entry.gpu_words.test(static_cast(word)); + } + if (all_gpu || !entry.flushed || Tegra::Nx2ListStats::t_reference) { + std::memcpy(out.data(), copy + offset, out.size()); + return; + } + device_memory.ReadBlockUnsafe(device_addr, out.data(), out.size()); + for (u64 byte = offset; byte < end;) { + const u64 next = (std::min)((byte / 4 + 1) * 4, end); + if (entry.gpu_words.test(static_cast(byte / 4))) { + std::memcpy(out.data() + (byte - offset), copy + byte, next - byte); + } + byte = next; + } + } +} + +template +void BufferCache

::Nx2MarkListWords(Nx2ListPage& entry, u64 begin, u64 end, bool set) { + if (begin >= end) { + return; + } + // Words GPU work may have written in part count whole; the host clears only whole words. + const u64 first = set ? begin / 4 : (begin + 3) / 4; + const u64 last = (std::min)(set ? (end + 3) / 4 : end / 4, Nx2ListPageWords); + for (u64 word = first; word < last; ++word) { + entry.gpu_words.set(static_cast(word), set); + } +} + +template +bool BufferCache

::Nx2HasListWords(const Nx2ListPage& entry, u64 begin, u64 end) { + const u64 last = (std::min)((end + 3) / 4, Nx2ListPageWords); + for (u64 word = begin / 4; word < last; ++word) { + if (entry.gpu_words.test(static_cast(word))) { + return true; + } + } + return false; +} + template void BufferCache

::Nx2NoteListSynced(DAddr page) { if constexpr (!IS_OPENGL) { std::scoped_lock lock{nx2_list_mutex}; Nx2ListPage& entry = nx2_list_pages[page]; entry.synced_writes = entry.writes; + entry.gpu_words.reset(); + entry.flushed = true; } } @@ -3420,6 +3973,8 @@ void BufferCache

::Nx2FlushListPages(std::span pages) { const auto it = nx2_list_pages.find(page); if (it != nx2_list_pages.end() && it->second.writes == writes) { it->second.synced_writes = writes; + it->second.gpu_words.reset(); + it->second.flushed = true; } } } @@ -3441,7 +3996,8 @@ void BufferCache

::Nx2NoteListHostWrite(DAddr device_addr, u64 size) { continue; } std::scoped_lock lock{nx2_list_mutex}; - const auto it = nx2_list_pages.find(page << page_bits); + const DAddr page_addr = page << page_bits; + const auto it = nx2_list_pages.find(page_addr); if (it == nx2_list_pages.end()) { continue; } @@ -3451,6 +4007,10 @@ void BufferCache

::Nx2NoteListHostWrite(DAddr device_addr, u64 size) { if (synced) { entry.synced_writes = entry.writes; } + Nx2MarkListWords(entry, (std::max)(device_addr, page_addr) - page_addr, + (std::min)(device_addr + size, page_addr + (DAddr{1} << page_bits)) - + page_addr, + false); } } } @@ -3696,7 +4256,8 @@ bool BufferCache

::SynchronizeBuffer(Buffer& buffer, DAddr device_addr, u32 si const auto add_upload = [&](DAddr start, DAddr end) { if (start == end) return; const u64 range_size = end - start; - ForgetUploadedEarlyLanded(start, range_size); // EDEN_NX2_READ_CHECK + NoteReadCheckEvent(start, range_size, ReadCheckEvent::Upload); // TEST + ForgetUploadedEarlyLanded(start, range_size); // TEST upload_copies.push_back(BufferCopy{ .src_offset = total_size_bytes, .dst_offset = start - buffer_start, @@ -3731,7 +4292,8 @@ bool BufferCache

::SynchronizeBuffer(Buffer& buffer, DAddr device_addr, u32 si u64 largest_copy = 0; DAddr buffer_start = buffer.CpuAddr(); for_each_upload_range([&](u64 device_addr_out, u64 range_size) { - ForgetUploadedEarlyLanded(device_addr_out, range_size); // EDEN_NX2_READ_CHECK + NoteReadCheckEvent(device_addr_out, range_size, ReadCheckEvent::Upload); // TEST + ForgetUploadedEarlyLanded(device_addr_out, range_size); // TEST copies.push_back(BufferCopy{ .src_offset = total_size_bytes, .dst_offset = device_addr_out - buffer_start, @@ -3830,8 +4392,8 @@ void BufferCache

::MappedUploadMemory([[maybe_unused]] Buffer& buffer, // Apply the staging offset copy.src_offset += upload_staging.offset; } - const bool can_reorder = runtime.CanReorderUpload(buffer, copies); - runtime.CopyBuffer(buffer, upload_staging.buffer, copies, true, can_reorder); + const auto order = runtime.DecideUploadOrder(buffer, copies); + runtime.CopyBuffer(buffer, upload_staging.buffer, copies, true, order); } } @@ -3859,6 +4421,7 @@ void BufferCache

::InlineMemoryImplementation(DAddr dest_address, size_t copy_ ClearDownload(dest_address, copy_size); gpu_modified_ranges.Subtract(dest_address, copy_size); MarkVertexBuffersStale(dest_address, copy_size); + NoteReadCheckEvent(dest_address, copy_size, ReadCheckEvent::Inline); // TEST BufferId buffer_id = FindBuffer(dest_address, static_cast(copy_size), false); auto& buffer = slot_buffers[buffer_id]; @@ -3873,8 +4436,8 @@ void BufferCache

::InlineMemoryImplementation(DAddr dest_address, size_t copy_ }}; u8* const src_pointer = upload_staging.mapped_span.data(); std::memcpy(src_pointer, inlined_buffer.data(), copy_size); - const bool can_reorder = runtime.CanReorderUpload(buffer, copies); - runtime.CopyBuffer(buffer, upload_staging.buffer, copies, true, can_reorder); + const auto order = runtime.DecideUploadOrder(buffer, copies); + runtime.CopyBuffer(buffer, upload_staging.buffer, copies, true, order); } else { buffer.ImmediateUpload(buffer.Offset(dest_address), inlined_buffer.first(copy_size)); } @@ -3914,6 +4477,7 @@ void BufferCache

::DownloadBuffersMemory(std::span buffer_ids) if (total_size_bytes == 0) { return; } + GpuWaitStats::AddDownloadBytes(total_size_bytes); const StutterTrace::SiteScope site{StutterTrace::Site::BufferDownload}; auto download_staging = runtime.DownloadStagingBuffer(total_size_bytes); for (Download& download : downloads) { @@ -3972,6 +4536,7 @@ void BufferCache

::DownloadBufferMemory(Buffer& buffer, DAddr device_addr, u64 if (total_size_bytes == 0) { return; } + GpuWaitStats::AddDownloadBytes(total_size_bytes); const StutterTrace::SiteScope site{StutterTrace::Site::BufferDownload}; // Guest memory is written once the host GPU ran the copies: the CPU writes collected meanwhile // keep their bytes too. diff --git a/src/video_core/buffer_cache/buffer_cache_base.h b/src/video_core/buffer_cache/buffer_cache_base.h index d5444a963d..eb2ba08f93 100644 --- a/src/video_core/buffer_cache/buffer_cache_base.h +++ b/src/video_core/buffer_cache/buffer_cache_base.h @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -44,10 +45,12 @@ #include "video_core/engines/maxwell_3d.h" #include "video_core/engines/kepler_compute.h" #include "video_core/engines/maxwell_3d.h" +#include "video_core/gpu_wait_stats.h" #include "video_core/memory_manager.h" +#include "video_core/nx2.h" #include "video_core/nx2_download_trace.h" -#include "video_core/nx2_async_readback.h" #include "video_core/nx2_fast_draw.h" +#include "video_core/nx2_read_trace.h" #include "video_core/rasterizer_download_area.h" #include "video_core/surface.h" #include "video_core/texture_cache/types.h" @@ -210,6 +213,8 @@ class BufferCache : public VideoCommon::ChannelSetupCaches FinishReadDownloads(DAddr device_addr, u64 size); + /// `mutex` itself (a caller holding it keeps it held while the host GPU finishes). + void FinishReadDownloads(DAddr device_addr, u64 size); /// A GPU write into the buffers over [device_addr, device_addr + size), after the read /// downloads taken so far: they still write their older bytes to guest memory, as synchronous @@ -306,21 +304,15 @@ class BufferCache : public VideoCommon::ChannelSetupCaches GetFlushArea(DAddr device_addr, u64 size); /// The core consumed the read traps of these pages for a read the texture cache answered: @@ -388,6 +380,10 @@ class BufferCache : public VideoCommon::ChannelSetupCaches ObtainCPUBuffer(DAddr gpu_addr, u32 size, ObtainBufferSynchronize sync_info, ObtainBufferOperation post_op); + + /// GPU work recorded now reads or writes these bytes, of a buffer obtained before (work set + /// up once that keeps reading them, as conditional rendering does). Creates no buffer. + void MarkUsed(DAddr device_addr, u64 size); void FlushCachedWrites(); /// Where the CPU writes into GPU-written memory that were collected but not applied yet @@ -434,21 +430,23 @@ class BufferCache : public VideoCommon::ChannelSetupCaches out, bool past_sync, u64* pending_tick = nullptr); /// Front thread, the split drained: the command list page was flushed to guest memory. void Nx2NoteListSynced(DAddr page); void Nx2FlushListPages(std::span pages); - /// CPU and inline writes: they reach guest memory, so a synced page stays synced, but its copy - /// no longer holds the page. + /// CPU, inline and copy engine writes of bytes guest memory holds: they reach guest memory, + /// so a synced page stays synced and the words they cover are guest memory's again, but the + /// page's copy no longer holds the page. void Nx2NoteListHostWrite(DAddr device_addr, u64 size); /// Return true when a CPU region is modified from the GPU @@ -629,7 +627,9 @@ class BufferCache : public VideoCommon::ChannelSetupCaches> ranges, @@ -778,6 +776,20 @@ class BufferCache : public VideoCommon::ChannelSetupCaches geometry_syncs{}; /// Counts DeleteBuffer calls: a buffer id older than this count may have been freed or reused. u64 buffer_deletions = 0; + + /// The last index buffer UpdateIndexBuffer resolved, and what it depends on. + struct IndexBufferMemo { + const void* channel{}; + const void* memory{}; + GPUVAddr begin{}; + GPUVAddr end{}; + u64 map_serial{~u64{0}}; + u64 deletions{}; + DAddr device_addr{}; + u32 found_size{}; ///< The size FindBuffer was asked for: the buffer holds at least this + BufferId buffer_id{}; + }; + IndexBufferMemo index_buffer_memo; struct GeometryStats { u64 syncs{}; u64 syncs_skipped{}; @@ -919,14 +931,10 @@ class BufferCache : public VideoCommon::ChannelSetupCaches current_buffer; - /// EDEN_NX2_READ_CHECK (1 in n read downloads): what a synchronous flush of - /// the read would have copied, copied again in the same submission, to compare guest memory - /// with once the download is written. + /// TEST (EDEN_NX2_READ_CHECK=n, 1 in n read downloads): what a synchronous flush of the read + /// would have copied, copied again in the same submission, to compare guest memory with once + /// the download is written. struct ReadCheck { - /// The bytes the read asked for, or the pages of an early download - DAddr read_begin{}; - DAddr read_end{}; - bool early{}; Async_Buffer staging{}; /// src_offset: the device address copied; dst_offset: where it is in `staging`. boost::container::small_vector copies; @@ -943,6 +951,17 @@ class BufferCache : public VideoCommon::ChannelSetupCaches own; + /// The downloads that brought `landed`, newest last, to explain an Outdated byte. + struct Origin { + DAddr begin{}; + DAddr end{}; + u64 tick{}; + u64 copied{}; ///< read_check_events before its copy + u64 landed{}; ///< read_check_events before it landed + bool early{}; + bool before_read{}; ///< Landed before this read came (early_landed), not kept + }; + boost::container::small_vector origins; }; struct ReadCheckTotals { u64 every{}; ///< Tests: check 1 in `every` read downloads, over EDEN_NX2_READ_CHECK @@ -950,31 +969,31 @@ class BufferCache : public VideoCommon::ChannelSetupCaches(VideoCore::AsyncReadback::Landing::Count)> - mismatched_by_landing{}; u64 uncovered{}; ///< Bytes a synchronous flush copies that no download brings u64 rewritten{}; ///< Kept bytes guest memory got anew after their download landed u64 logged{}; ///< Error lines so far (at most 16) u64 logged_rewrites{}; ///< Lines about rewritten bytes so far (at most 16) - u64 reads_recorded{}; ///< Checks recorded with a read's download - u64 alone_recorded{}; ///< Checks recorded alone (other downloads bring the read's bytes) - u64 early_recorded{}; ///< Checks recorded with an early download u64 last_log{}; + u64 reads_recorded{}; ///< Read downloads recorded with a check + u64 alone_recorded{}; ///< Checks recorded alone (their read's bytes on their way) + u64 early_recorded{}; ///< Early downloads recorded with a check + u64 early_left_out{}; ///< Early downloads not recorded: past their bounds + u64 early_served{}; ///< Reads an early download served, made to flush for the check }; ReadCheckTotals read_check{}; static u64 ReadCheckEvery() { - return VideoCore::AsyncReadback::ReadCheckEvery(); + static const u64 every = [] { + const char* value = std::getenv("EDEN_NX2_READ_CHECK"); + if (value == nullptr) { + return u64{0}; + } + return (std::max)(u64{1}, static_cast(std::strtoull(value, nullptr, 10))); + }(); + return every; } - void CheckReadDownload(const ReadCheck& check, const Common::RangeSet& cpu_written, - VideoCore::AsyncReadback::Landing landing); [[nodiscard]] bool ReadChecks() const { return read_check.every != 0 || ReadCheckEvery() != 0; } - /// A download landed `written`, but for the guest CPU writes collected then (`cpu_written`): - /// what guest memory holds now over a later check's `kept`. - void NoteLandedBytes(ReadCheck& check, const Common::RangeSet& written, - const Common::RangeSet& cpu_written); /// A DownloadMemoryForRead: its copies, and what is left of them to do once the host GPU has /// run them. @@ -990,32 +1009,91 @@ class BufferCache : public VideoCommon::ChannelSetupCaches clean_ranges; - std::unique_ptr check; ///< EDEN_NX2_READ_CHECK + std::unique_ptr check; ///< TEST: EDEN_NX2_READ_CHECK /// Recorded right after a GPU write, before any read (RecordEarlyDownloads) bool early{}; + ReadTrace::Copy trace{}; ///< TEST: EDEN_NX2_READ_TRACE + u64 events_before{}; ///< TEST (EDEN_NX2_READ_CHECK): read_check_events before it + }; + /// A download landed its write_ranges, but for the guest CPU writes collected then + /// (`cpu_written`): what guest memory holds now over a later check's `kept`. + void NoteLandedBytes(ReadCheck& check, const ReadDownload& download, + const Common::RangeSet& cpu_written); + /// TEST (EDEN_NX2_READ_CHECK): compares guest memory with `download`'s check copy, right after + /// the download landed. + void CheckReadDownload(const ReadDownload& download, + const Common::RangeSet& cpu_written); + /// TEST (EDEN_NX2_READ_CHECK): where a difference the check found comes from. + void ExplainReadCheckDifference(const ReadDownload& download, DAddr address, u64 size, u8 found, + u8 wanted); + /// TEST (EDEN_NX2_READ_CHECK): what changed the GPU's copy of an Outdated byte, which an + /// earlier download brought: the writes, guest writes and uploads over it since that copy. + void ExplainReadCheckOutdated(const ReadDownload& download, DAddr address, u64 size); + enum class ReadCheckEvent : u8 { + GpuWrite, ///< MarkWrittenBuffer or DMACopy, with its GpuWaitStats write site + Unrecorded, ///< A GPU write the cache records only as outdating downloads (SyncValues) + DmaClear, + GuestWrite, ///< WriteMemory: a guest CPU write, or the GPU's side through guest memory + Upload, ///< SynchronizeBuffer + Inline, ///< InlineMemoryImplementation + }; + struct ReadCheckEventEntry { + DAddr begin{}; + DAddr end{}; + u64 sequence{}; ///< 1 for the first event + ReadCheckEvent kind{}; + GpuWaitStats::WriteSite site{}; }; + /// The last ReadCheckEventRing events, sized by the first: a cache without the check has none. + std::vector read_check_events; + u64 read_check_event_count{}; + static constexpr size_t ReadCheckEventRing = 4096; + /// TEST (EDEN_NX2_READ_CHECK): something changed the GPU's copy or guest memory there. + void NoteReadCheckEvent(DAddr device_addr, u64 size, ReadCheckEvent kind, + GpuWaitStats::WriteSite site = GpuWaitStats::WriteSite::Other) { + if (!ReadChecks() || size == 0 || (read_downloads.empty() && early_landed.empty())) { + return; + } + if (read_check_events.empty()) [[unlikely]] { + read_check_events.resize(ReadCheckEventRing); + } + if (kind == ReadCheckEvent::Unrecorded && read_check_event_count != 0) { + // MarkWrittenBuffer outdates the downloads after noting its write. + const ReadCheckEventEntry& last = + read_check_events[(read_check_event_count - 1) % read_check_events.size()]; + if (last.kind == ReadCheckEvent::GpuWrite && last.begin == device_addr && + last.end == device_addr + size) { + return; + } + } + ++read_check_event_count; + read_check_events[(read_check_event_count - 1) % read_check_events.size()] = { + device_addr, device_addr + size, read_check_event_count, kind, site}; + } /// Oldest first, and written to guest memory in that order: none lands over newer bytes. std::deque read_downloads; /// All the read downloads' clean_ranges (they don't overlap): GPU-modified bytes already on /// their way, which later reads wait for rather than download again. Common::RangeSet read_download_ranges; - /// EDEN_NX2_READ_CHECK: the bytes early downloads brought to a page's guest memory, and + /// TEST (EDEN_NX2_READ_CHECK): the bytes early downloads brought to a page's guest memory, and /// what guest memory held there as they landed, until newer downloaded bytes or values replace /// them. The read they serve compares them with the GPU's bytes then: a GPU write in between /// that took nothing out of them shows. struct EarlyLanded { Common::RangeSet bytes; std::vector as_landed; ///< The page, DEVICE_PAGESIZE bytes + /// The downloads that landed them, newest last (at most 8): a check's Origin. + boost::container::small_vector from; }; std::unordered_map early_landed; - /// EDEN_NX2_READ_CHECK: an upload sends guest memory's bytes to the buffer. Where they + /// TEST (EDEN_NX2_READ_CHECK): an upload sends guest memory's bytes to the buffer. Where they /// are no longer what an early download landed, the guest wrote there since, and landed bytes /// are no longer GPU-modified, so that write may never have reached WriteMemory. The buffer /// then holds it while guest memory may hold the landed value again: no read compares those /// bytes, as a synchronous flush would not download them. void ForgetUploadedEarlyLanded(DAddr device_addr, u64 size) { - if (!early_landed.empty()) { + if (!early_landed.empty()) [[unlikely]] { ForgetRewrittenEarlyLanded(device_addr, size); } } @@ -1030,6 +1108,27 @@ class BufferCache : public VideoCommon::ChannelSetupCaches trace_writes{}; + size_t trace_write_count{}; + u64 trace_sequence{}; + bool trace_marks_pending{}; + /// The GPU thread recorded a draw or dispatch: a device timestamp after it dates its writes + /// to pages the guest reads back. + void TraceWorkRecorded(bool dispatch); + void TraceWriteRange(DAddr device_addr, u64 size); + void TraceFlushWriteMarks(); + /// The reading thread's read waits for `download` (`mutex` held). + void TraceReadCopy(ReadTrace::Read& read, const ReadDownload& download, DAddr device_addr, + u64 size) const; + /// A page the guest CPU read GPU writes from. Switch 2 titles read compute results back every /// frame, milliseconds after the dispatch that wrote them: a download recorded at the read /// waits for all the GPU work recorded in between. Pages read after each of their writes get @@ -1041,6 +1140,7 @@ class BufferCache : public VideoCommon::ChannelSetupCaches read_back_pages; ///< Sorted by page - /// NoteReportPending's results not yet written, oldest first. struct PendingReport { DAddr addr; u64 size; - bool rewritten; ///< A GPU write over it since the report + bool rewritten; }; std::vector pending_reports; /// Pages to download once the work writing them is recorded. @@ -1100,8 +1207,9 @@ class BufferCache : public VideoCommon::ChannelSetupCaches().DownloadStagingBuffer(size_t{}, true)); + static constexpr size_t Nx2ListPageWords = (size_t{1} << Tegra::Nx2WrittenPageBits) / 4; struct Nx2ListPage { - u64 writes{}; ///< GPU writes noted to the page + u64 writes{}; ///< GPU and host writes noted to the page u64 copy_writes{}; ///< `writes` when the copy was made: the copy is current while equal u64 copy_tick{}; ///< GPU tick that completes the copy /// `writes` when the front last flushed the page: guest memory holds it while equal @@ -1114,8 +1222,25 @@ class BufferCache : public VideoCommon::ChannelSetupCaches gpu_words; + bool flushed{}; std::optional copy; }; + /// Sets (`set`) or clears the words of `entry.gpu_words` within [begin, end) bytes of its + /// page: every word the bytes touch when setting, only the words they cover when clearing. + static void Nx2MarkListWords(Nx2ListPage& entry, u64 begin, u64 end, bool set); + /// Whether GPU work wrote any word of [begin, end) bytes of the page guest memory lacks. + [[nodiscard]] static bool Nx2HasListWords(const Nx2ListPage& entry, u64 begin, u64 end); + /// Fills `out` with the page's bytes from `offset` (`device_addr`) on: the words of + /// `entry.gpu_words` from its copy, the others from guest memory. + void Nx2ReadListWords(const Nx2ListPage& entry, DAddr device_addr, u64 offset, + std::span out); std::mutex nx2_list_mutex; ///< nx2_list_pages: the split's front reads the copies std::unordered_map nx2_list_pages; /// Copies replaced while the GPU may still write them: freed once it is past their tick. diff --git a/src/video_core/buffer_cache/uniform_stream_reuse.h b/src/video_core/buffer_cache/uniform_stream_reuse.h index 7bd76b979c..defc07b55e 100644 --- a/src/video_core/buffer_cache/uniform_stream_reuse.h +++ b/src/video_core/buffer_cache/uniform_stream_reuse.h @@ -31,13 +31,24 @@ concept StreamUniformReusable = requires(R& runtime, u32 size) { { runtime.CurrentTick() } -> std::convertible_to; }; +[[nodiscard]] inline bool UniformStreamReuseSwitch() { + const char* const value = std::getenv("EDEN_NX2_UBOREUSE"); + return value == nullptr || value[0] != '0'; +} + /// EDEN_NX2_UBOREUSE=0 turns the reuse off (every small uniform buffer is copied, as before). +/// A plain flag: checked for every uniform buffer of every draw, where a function's static is +/// checked for its initialization on every call (MSVC reads a thread-local for it). +inline bool g_uniform_stream_reuse = UniformStreamReuseSwitch(); + +/// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads +/// nx2_switches.txt, so a switch set only there takes effect from that point on. +inline void ReadUniformStreamReuseSwitch() { + g_uniform_stream_reuse = UniformStreamReuseSwitch(); +} + [[nodiscard]] inline bool UniformStreamReuseEnabled() noexcept { - static const bool enabled = [] { - const char* const value = std::getenv("EDEN_NX2_UBOREUSE"); - return value == nullptr || value[0] != '0'; - }(); - return enabled; + return g_uniform_stream_reuse; } template @@ -66,6 +77,7 @@ class UniformStreamReuse { std::array, NumStages> slots{}; std::array scratch{}; + u64 rebinds{}; ///< TEST: allocations bound again (EDEN_NX2_GPU_PROFILE=2 counts them) }; template @@ -98,25 +110,34 @@ template } Reuse& reuse = UniformStreamReuseState(); typename Reuse::Slot& slot = reuse.slots[stage][index]; - u8* const scratch = reuse.scratch.data(); - device_memory.ReadBlockUnsafe(device_addr, scratch, size); - - const bool same_bytes = - slot.size == size && std::memcmp(slot.bytes.data(), scratch, size) == 0; + // Guest memory in one piece of host memory (most of these buffers lie within a page) is + // compared where it is; other ranges are gathered into scratch first. + const u8* guest = nullptr; + if constexpr (requires { device_memory.GetSpan(device_addr, size_t{size}); }) { + guest = device_memory.GetSpan(device_addr, size_t{size}); + } + if (guest == nullptr) { + device_memory.ReadBlockUnsafe(device_addr, reuse.scratch.data(), size); + guest = reuse.scratch.data(); + } + const bool same_bytes = slot.size == size && std::memcmp(slot.bytes.data(), guest, size) == 0; slot.device_addr = device_addr; if (same_bytes && slot.tick == runtime.CurrentTick()) { runtime.RebindStreamUniform(slot.ref, size); + ++reuse.rebinds; return true; } + if (!same_bytes) { + // The slot takes the bytes first and the stream gets the slot's: the guest may write its + // memory meanwhile, and the slot must hold what its allocation holds. + slot.size = size; + std::memcpy(slot.bytes.data(), guest, size); + } const std::span span = runtime.BindMappedUniformBuffer(stage, binding_index, size); - std::memcpy(span.data(), scratch, size); + std::memcpy(span.data(), slot.bytes.data(), size); slot.ref = runtime.LastStreamUniformRef(); // After the allocation: the tick the stream region is tagged with is the one current now. slot.tick = runtime.CurrentTick(); - if (!same_bytes) { - slot.size = size; - std::memcpy(slot.bytes.data(), scratch, size); - } return true; } diff --git a/src/video_core/buffer_cache/upload_order.h b/src/video_core/buffer_cache/upload_order.h new file mode 100644 index 0000000000..d5576cacfe --- /dev/null +++ b/src/video_core/buffer_cache/upload_order.h @@ -0,0 +1,157 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include + +#include "common/common_types.h" + +namespace VideoCommon { + +/// Where an upload into a buffer runs. +struct UploadOrder { + /// In the submission's upload command buffer, ahead of all the work of its main one. If not, + /// at its place in the main command buffer, between barriers, after the work recorded before + /// it (a render pass in progress ends first). + bool reorder{}; + /// When reordered: the newest earlier submission whose work used the buffer, 0 if none. The + /// upload command buffer begins with a barrier on earlier work unless the host GPU finished + /// that submission by the time the upload command buffer is recorded. + u64 prior_tick{}; +}; + +/// The bytes of a buffer that GPU work recorded in a submission uses (reads or writes), by the +/// tick of the submission. Byte ranges are kept for the newest submission only: the ones before +/// it collapse into the tick of the newest of them, which no byte's last use is later than. +class SubmissionUses { +public: + /// Disjoint ranges are merged with their neighbors beyond this many: the uses then cover + /// more bytes than the work used, never fewer. + static constexpr size_t MaxRanges = 1024; + + /// Work recorded in submission `tick` uses [offset, offset + size). Ticks never go back. + void Track(u64 offset, u64 size, u64 tick) noexcept { + if (size == 0) { + return; + } + if (tick != current_tick) { + if (tick < current_tick) { + previous_tick = (std::max)(previous_tick, tick); + return; + } + if (!ranges.empty()) { + previous_tick = current_tick; + } + ranges.clear(); + recent.fill(Range{}); + current_tick = tick; + } + const u64 begin = offset; + const u64 end = offset + size; + // Draws bind the same ranges again and again. Ranges only grow until the tick changes, + // so one that covered these bytes still does. + for (const Range& range : recent) { + if (range.begin <= begin && end <= range.end) { + return; + } + } + recent[recent_cursor++ % recent.size()] = Range{begin, end}; + // The first range that ends at or after begin: it overlaps or touches [begin, end), or + // lies after it. + auto first = + std::lower_bound(ranges.begin(), ranges.end(), begin, + [](const Range& range, u64 value) { return range.end < value; }); + if (first != ranges.end() && first->begin <= begin && end <= first->end) { + return; + } + u64 merged_begin = begin; + u64 merged_end = end; + auto last = first; + while (last != ranges.end() && last->begin <= end) { + merged_begin = (std::min)(merged_begin, last->begin); + merged_end = (std::max)(merged_end, last->end); + ++last; + } + if (first == last) { + ranges.insert(first, Range{merged_begin, merged_end}); + } else { + *first = Range{merged_begin, merged_end}; + ranges.erase(first + 1, last); + } + if (ranges.size() > MaxRanges) { + Coarsen(); + } + } + + /// Whether work recorded in submission `tick` uses a byte of [offset, offset + size). + [[nodiscard]] bool IsUsed(u64 offset, u64 size, u64 tick) const noexcept { + if (size == 0 || tick != current_tick || ranges.empty()) { + return false; + } + const u64 begin = offset; + const u64 end = offset + size; + // The first range that ends after begin. + const auto it = + std::upper_bound(ranges.begin(), ranges.end(), begin, + [](u64 value, const Range& range) { return value < range.end; }); + return it != ranges.end() && it->begin < end; + } + + /// The newest submission before `tick` whose work used a byte of the buffer, 0 if none did. + [[nodiscard]] u64 NewestTickBefore(u64 tick) const noexcept { + // current_tick is 0 until the first use. Asked about the past (current_tick > tick), the + // newest use known is still a bound. + return current_tick == tick ? previous_tick : current_tick; + } + + /// How many disjoint ranges the newest submission's work uses (for tests). + [[nodiscard]] size_t RangeCount() const noexcept { + return ranges.size(); + } + +private: + struct Range { + u64 begin; + u64 end; + }; + + /// Merges each pair of neighbors, gap included. + void Coarsen() noexcept { + size_t out = 0; + for (size_t in = 0; in < ranges.size(); in += 2) { + const u64 merged_end = in + 1 < ranges.size() ? ranges[in + 1].end : ranges[in].end; + ranges[out++] = Range{ranges[in].begin, merged_end}; + } + ranges.resize(out); + recent.fill(Range{}); + } + + /// Sorted, disjoint, and never touching: neighbors merge. + std::vector ranges; + /// Ranges tracked lately in current_tick, as given. + std::array recent{}; + u32 recent_cursor{}; + u64 current_tick{}; + u64 previous_tick{}; +}; + +/// Where an upload that writes `copies` (dst_offset, size) of a buffer with these uses runs when +/// it is recorded in submission `tick`: in order when work recorded before it in the submission +/// uses a byte it writes, otherwise in the upload command buffer, after the work of the earlier +/// submissions that used the buffer. The caller then tracks the copies as uses: an upload of the +/// same bytes later in the submission stays in order behind this one. +template +[[nodiscard]] UploadOrder DecideUploadOrder(const SubmissionUses& uses, const Copies& copies, + u64 tick) { + for (const auto& copy : copies) { + if (uses.IsUsed(copy.dst_offset, copy.size, tick)) { + return UploadOrder{}; + } + } + return UploadOrder{.reorder = true, .prior_tick = uses.NewestTickBefore(tick)}; +} + +} // namespace VideoCommon diff --git a/src/video_core/control/scheduler.cpp b/src/video_core/control/scheduler.cpp index 62b5e62eda..4d2a4c639e 100644 --- a/src/video_core/control/scheduler.cpp +++ b/src/video_core/control/scheduler.cpp @@ -14,6 +14,7 @@ #include "video_core/control/scheduler.h" #include "video_core/gpu.h" #include "video_core/gpu_split.h" +#include "video_core/nx2_profile.h" namespace Tegra::Control { @@ -21,10 +22,12 @@ void Scheduler::BindChannel(GPU& gpu, s32 bind_id) { // With the GPU thread split the rasterizer's channel switches with the methods around it, on // the back thread (deferred outside the lock: the back takes it to bind). if (auto& split = VideoCommon::GpuSplit::Get(); split.Active()) { - split.Defer([this, gpu_ptr = &gpu, bind_id] { - std::scoped_lock lk{scheduling_guard}; - gpu_ptr->BindChannel(bind_id); - }); + split.Defer( + [this, gpu_ptr = &gpu, bind_id] { + std::scoped_lock lk{scheduling_guard}; + gpu_ptr->BindChannel(bind_id); + }, + VideoCommon::GpuSplit::Writes::Noted); return; } std::scoped_lock lk{scheduling_guard}; @@ -68,9 +71,23 @@ bool Scheduler::ResumePending(GPU& gpu) { if (!dma_pusher.system.IsPoweredOn()) { return any_progress; } - if (!dma_pusher.HasPendingAcquire() || !dma_pusher.CanResumeAcquire()) { + if (!dma_pusher.HasPendingAcquire()) { continue; } + // TEST: the poll apart from the resumed channel's commands (EDEN_NX2_GPU_PROFILE). + bool can_resume; + { + VideoCommon::Nx2Profile::Scope scope{VideoCommon::Nx2Profile::Zone::AcquirePoll}; + can_resume = dma_pusher.CanResumeAcquire(); + } + if (!can_resume) { + continue; + } + // The back ran the release: it may be out of work by the time the channel's next + // commands would otherwise go to it. + if (auto& split = VideoCommon::GpuSplit::Get(); split.Active()) { + split.PublishSoon(); + } BindChannel(gpu, state->bind_id); dma_pusher.DispatchCalls(); progress = true; diff --git a/src/video_core/dma_pusher.cpp b/src/video_core/dma_pusher.cpp index fbe3e07086..3e92b8477c 100644 --- a/src/video_core/dma_pusher.cpp +++ b/src/video_core/dma_pusher.cpp @@ -9,6 +9,8 @@ #include #include #include +#include +#include #include #include #include @@ -21,7 +23,9 @@ #include "video_core/dma_pusher.h" #include "video_core/command_fetch_order.h" #include "video_core/control/channel_state.h" +#include "video_core/engines/engine_upload.h" #include "video_core/engines/kepler_compute.h" +#include "video_core/engines/kepler_memory.h" #include "video_core/engines/maxwell_3d.h" #include "video_core/engines/maxwell_dma.h" #include "video_core/gpu.h" @@ -30,6 +34,11 @@ #include "video_core/memory_manager.h" #include "video_core/nx2.h" #include "video_core/nx2_cmdpage_diag.h" +#include "video_core/nx2_drain_check.h" +#include "video_core/nx2_entry_check.h" +#include "video_core/nx2_list_indirect.h" +#include "video_core/nx2_list_indirect_stats.h" +#include "video_core/nx2_list_stats.h" #include "video_core/rasterizer_interface.h" namespace Tegra { @@ -37,6 +46,68 @@ namespace Tegra { constexpr u32 MacroRegistersStart = 0xE00; constexpr u32 ComputeInline = 0x6D; +namespace { + +/// The split's position after the last copy engine method that may write memory (the front's). +u64 g_last_copy_writer{}; + +/// Whether copy engine work sent so far may still be queued on the split's back. A new session +/// starts the split's positions over: a position past the current one is from an older session. +[[nodiscard]] bool CopyWritersPending(const VideoCommon::GpuSplit& split) { + if (g_last_copy_writer > split.Position()) { + g_last_copy_writer = 0; + } + return !split.Ran(g_last_copy_writer); +} + +/// EDEN_NX2_LIST_STATS and _VERIFY: the front read `words` of a command list at `address` +/// (`device`) on a GPU-written page, as `where` tells; see Nx2ListStats::NoteRead. +void NoteListRead(MemoryManager& memory, s32 channel, GPUVAddr address, u64 device, + std::span words, VideoCore::RasterizerInterface::GpuWrittenList where, + bool after_sync, u64 backlog, u64 unpublished) { + using GpuWrittenList = VideoCore::RasterizerInterface::GpuWrittenList; + using Nx2ListStats::Served; + std::optional copy_equal; + if (Nx2ListStats::g_enabled && where == GpuWrittenList::Copied) { + std::vector guest(words.size()); + memory.ReadBlockUnsafe(address, guest.data(), guest.size()); + copy_equal = std::ranges::equal(guest, words); + } + const Served result = where == GpuWrittenList::Copied ? Served::CopyReady + : where == GpuWrittenList::Synced ? Served::PageSynced + : Served::Unknown; + constexpr u64 page_mask = Nx2ListStats::PageBytes - 1; + Nx2ListStats::NoteRead(device & ~page_mask, static_cast(device & page_mask), + static_cast(words.size()), result, after_sync, copy_equal, backlog, + unpublished, channel); +} + +static_assert(DmaPusher::DmaRegisterCount == Engines::MaxwellDMA::RegisterCount); +static_assert(DmaPusher::InlineToMemoryRegisterCount == Engines::KeplerMemory::Regs::NUM_REGS); + +/// What the back writes running `method` of an engine the front sends methods to as they are +/// (GpuSplit::Writes). A method the engine only keeps for later (its method sink) writes nothing +/// yet. Copy launches write the caches where they read GPU-written memory; what they copy to and +/// release, and what inline-to-memory uploads write, DmaPusher::NoteEngineWrites places. +VideoCommon::GpuSplit::Writes EngineWrites(const Engines::EngineInterface& engine, + Engines::EngineTypes type, u32 method) { + using Writes = VideoCommon::GpuSplit::Writes; + if (!engine.execution_mask[method]) { + return Writes::Noted; + } + switch (type) { + case Engines::EngineTypes::MaxwellDMA: + return Writes::Caches; + case Engines::EngineTypes::KeplerMemory: + return Writes::Noted; + default: + // Blits and the engines without a front: where they write, the front does not know. + return Writes::Unplaced; + } +} + +} // Anonymous namespace + bool DmaPusher::IsMemoryWriter(const Engines::EngineInterface& engine, Engines::EngineTypes type, u32 method) { if (type == Engines::EngineTypes::KeplerCompute) { @@ -60,10 +131,8 @@ DmaPusher::~DmaPusher() = default; bool DmaPusher::CanResumeAcquire() { if (pending_list_tick) { - VideoCommon::Count(VideoCommon::g_acquire_stats.list_polls); return rasterizer->IsGpuTickDone(*pending_list_tick); } - VideoCommon::Count(VideoCommon::g_acquire_stats.acquire_polls); return puller.CanResumeAcquire(*this); } @@ -76,10 +145,13 @@ void DmaPusher::DispatchCalls() { } } if (auto& split = VideoCommon::GpuSplit::Get(); split.Active()) { - split.Defer([gpu = &system.GPU()] { - gpu->FlushCommands(); - gpu->OnCommandListEnd(); - }); + // Submits, and releases the fences done (downloads where the GPU wrote). + split.Defer( + [gpu = &system.GPU()] { + gpu->FlushCommands(); + gpu->OnCommandListEnd(); + }, + VideoCommon::GpuSplit::Writes::Caches); split.Publish(); return; } @@ -98,7 +170,7 @@ enum class LiveReadReason : u32 { Count, }; -/// EDEN_NX2_LIVE_READ_CHECK: compares the words an entry was parsed from with its submit-time copy. +/// EDEN_NX2_LIVE_READ_CHECK: compare parsed words with their submit-time copy. void CheckLiveRead(LiveReadReason reason, const CommandListHeader& header, std::span parsed, const CommandHeader* copy) { constexpr size_t count = static_cast(LiveReadReason::Count); @@ -112,7 +184,7 @@ void CheckLiveRead(LiveReadReason reason, const CommandListHeader& header, u32 first = ~0U, words = 0; for (u32 i = 0; i < parsed.size(); ++i) { if (copy[i].argument != parsed[i].argument) { - first = std::min(first, i); + first = (std::min)(first, i); ++words; } } @@ -134,7 +206,7 @@ void CheckLiveRead(LiveReadReason reason, const CommandListHeader& header, } } -} // Anonymous namespace +} // namespace bool DmaPusher::Step() { if (!ib_enable || dma_pushbuffer.empty()) { @@ -164,12 +236,16 @@ bool DmaPusher::Step() { // as this frame's commands (methods on unbound subchannels, garbage QMDs, then a channel // waiting forever on a release it never got). The acquire is released now, so the commands // are written and the memory not yet reused: copy them here. EDEN_NX2_RESNAPSHOT=0: off. + // The current list's entries before this one ran already, and an entry the acquire suspended + // inside reads its remaining words from memory anyway (CanUseSubmittedCommands). if (ResnapshotEnabled() && suspensions.load(std::memory_order_relaxed) != suspensions_before) { const u64 now = suspensions.load(std::memory_order_acquire); + size_t first_entry = dma_pushbuffer_subindex + (command_resume_offset != 0 ? 1 : 0); for (CommandList& list : dma_pushbuffer) { if (!list.submitted_offsets.empty()) { - RefreshSnapshot(list, now); + RefreshSnapshot(list, now, first_entry); } + first_entry = 0; } } @@ -228,10 +304,13 @@ bool DmaPusher::Step() { auto* const engine_state = subchannels[dma_state.subchannel]; if (auto& split = VideoCommon::GpuSplit::Get(); split.Active()) { // The caches behind IsMemoryDirty belong to the split's back thread. - split.Defer([engine_state, memory = &memory_manager, address = GPUVAddr{dma_state.dma_get}, - bytes = words * sizeof(u32)] { - engine_state->current_dirty = memory->IsMemoryDirty(address, bytes); - }, VideoCommon::GpuSplit::FastDrawNeutral); + split.Defer( + [engine_state, memory = &memory_manager, + address = GPUVAddr{dma_state.dma_get}, bytes = words * sizeof(u32)] { + engine_state->current_dirty = memory->IsMemoryDirty(address, bytes); + }, + VideoCommon::GpuSplit::Writes::Noted, + VideoCommon::GpuSplit::FastDrawNeutral); } else { engine_state->current_dirty = memory_manager.IsMemoryDirty(dma_state.dma_get, words * sizeof(u32)); } @@ -276,11 +355,28 @@ bool DmaPusher::Step() { // queued before it. The GPU fetches the list after that work only past a host-level wait // (NoteSync): later work is not ordered before the fetch, and the list does not wait for // it (nor for draws, which write memory too; draw-written lists wait for the last sync). + // Lists that begin between methods don't wait for compute work either: they are command + // buffers the CPU writes. The words GPU work writes for the front to fetch are the + // parameters of a method begun in the entry before (an indirect dispatch's or draw's, + // as in Ocarina of Time and title 1), which begin inside its data. An entry resumed after + // an acquire resumes after a puller method, which no parameter list holds. The copy + // engine can copy GPU-written words anywhere: while its work is queued, every list waits. + // The lists that did not wait are checked on the back (EDEN_NX2_DRAIN_CHECK). + const bool command_entry = dma_state.method_count == 0; + const bool waits_for_writers = + !command_entry || CopyWritersPending(VideoCommon::GpuSplit::Get()); if (!gpu_written && VideoCommon::GpuSplit::Get().Active()) { - VideoCommon::GpuSplit::Get().WaitForSyncedWriters(); + if (waits_for_writers) { + VideoCommon::GpuSplit::Get().WaitForSyncedWriters(); + } + // A writer may have marked the page since the first check without making us wait. gpu_written = IsGpuWrittenCommands(dma_state.dma_get, size_t{header.size} * sizeof(u32)); } + if (VideoCommon::Nx2DrainCheck::g_enabled && !gpu_written && waits_for_writers) + [[unlikely]] { + CheckUnwaitedWriters(dma_state.dma_get, size_t{header.size} * sizeof(u32)); + } // Resume and GPU-written entries need live words, even after a flush cleared dirty state. const bool submitted = (command_list.snapshot_suspensions & 1) == 0 && command_list.snapshot_suspensions == @@ -293,10 +389,16 @@ bool DmaPusher::Step() { gpu_written, use_safe && memory_manager.IsMemoryDirty(dma_state.dma_get, header.size * sizeof(u32))); - // An entry parsed from memory instead of its submit-time copy is the way a command - // ring the game already refilled gets read as this frame's commands (garbage methods, - // unbound subchannels, render targets of absurd size). EDEN_NX2_LIVE_READ_CHECK counts - // how often the words parsed differ from the copy, by the reason the copy was skipped. + if (Nx2EntryCheck::Any() && !gpu_written && !use_safe) [[unlikely]] { + Nx2EntryCheck::NoteEntry( + *this, header, + submitted ? std::span{command_list.submitted_words.data() + + command_list.submitted_offsets + [dma_pushbuffer_subindex], + header.size} + : std::span{}, + !waits_for_writers, command_entry && waits_for_writers); + } static const bool live_read_check = std::getenv("EDEN_NX2_LIVE_READ_CHECK") != nullptr; const CommandHeader* const submitted_copy = live_read_check && !submitted && @@ -307,9 +409,9 @@ bool DmaPusher::Step() { command_list.submitted_offsets[dma_pushbuffer_subindex] : nullptr; const LiveReadReason live_read_reason = - submitted_copy == nullptr ? LiveReadReason::Count - : command_resume_offset != 0 ? LiveReadReason::Resumed - : (command_list.snapshot_suspensions & 1) != 0 ? LiveReadReason::SuspendedCopy + submitted_copy == nullptr ? LiveReadReason::Count + : command_resume_offset != 0 ? LiveReadReason::Resumed + : (command_list.snapshot_suspensions & 1) != 0 ? LiveReadReason::SuspendedCopy : command_list.snapshot_suspensions != suspensions.load(std::memory_order_acquire) ? LiveReadReason::SuspensionsChanged : gpu_written ? LiveReadReason::GpuWritten @@ -332,6 +434,12 @@ bool DmaPusher::Step() { Nx2CmdPageDiag::CountEntry(Nx2CmdPageDiag::Path::Safe); command_resume_offset += ProcessCommands(commands.subspan(command_resume_offset), command_resume_offset); + } else if (gpu_written && TakeGpuWrittenListWords(header)) { + // The words stay on the host GPU: the call takes their placeholders (not recorded + // for EDEN_NX2_PB_CHECK, which would find the entry's memory changed). + const std::span commands{command_headers.data(), header.size}; + command_resume_offset += + ProcessCommands(commands.subspan(command_resume_offset), command_resume_offset); } else if (bool pending = false; ReadGpuWrittenCommands(dma_state.dma_get, static_cast(header.size), &pending) || @@ -575,11 +683,15 @@ void DmaPusher::CallMethod(u32 argument) { return; } const auto engine_type = subchannel_type[dma_state.subchannel]; + ShadowEngineMethod(engine_type, dma_state.method, argument); const bool writer = split.Active() && IsMemoryWriter(*subchannel, engine_type, dma_state.method); SCOPE_EXIT { if (writer) { split.NoteWriter(); + if (engine_type == Engines::EngineTypes::MaxwellDMA) { + g_last_copy_writer = split.Position(); + } } }; if (split.Active()) { @@ -603,10 +715,10 @@ void DmaPusher::CallMethod(u32 argument) { } } split.Method(subchannel, dma_state.method, argument, dma_state.is_last_call, - WantsSegment() ? &segment : nullptr); - if (engine_type == Engines::EngineTypes::MaxwellDMA) { - NoteDmaSemaphore(dma_state.method, argument); - } + WantsSegment() ? &segment : nullptr, + EngineWrites(*subchannel, engine_type, dma_state.method)); + split.TagUnplaced(VideoCommon::UnplacedKind::Engine); // TEST + NoteEngineWrites(engine_type, dma_state.method); return; } if (!subchannel->execution_mask[dma_state.method]) { @@ -641,6 +753,9 @@ void DmaPusher::CallMultiMethod(const u32* base_start, u32 num_methods) { SCOPE_EXIT { if (writer) { split.NoteWriter(); + if (engine_type == Engines::EngineTypes::MaxwellDMA) { + g_last_copy_writer = split.Position(); + } } }; if (split.Active()) { @@ -664,14 +779,26 @@ void DmaPusher::CallMultiMethod(const u32* base_start, u32 num_methods) { } } split.MultiMethod(subchannel, dma_state.method, base_start, num_methods, - dma_state.method_count, WantsSegment() ? &segment : nullptr); - if (engine_type == Engines::EngineTypes::MaxwellDMA) { - for (u32 i = 0; i < num_methods; ++i) { - NoteDmaSemaphore(dma_state.method, base_start[i]); + dma_state.method_count, WantsSegment() ? &segment : nullptr, + EngineWrites(*subchannel, engine_type, dma_state.method)); + split.TagUnplaced(VideoCommon::UnplacedKind::Engine); // TEST + // The back runs them one by one: a copy engine launch each. + const bool launches = engine_type == Engines::EngineTypes::MaxwellDMA && + dma_state.method == Engines::MaxwellDMA::LaunchDmaMethod; + for (u32 i = 0; i < num_methods; ++i) { + ShadowEngineMethod(engine_type, dma_state.method, base_start[i]); + if (launches) { + NoteEngineWrites(engine_type, dma_state.method); } } + if (!launches && num_methods != 0) { + NoteEngineWrites(engine_type, dma_state.method); + } return; } + for (u32 i = 0; i < num_methods; ++i) { + ShadowEngineMethod(engine_type, dma_state.method, base_start[i]); + } VideoCommon::Nx2Profile::Scope method_scope{VideoCommon::Nx2Profile::Zone::Method}; subchannel->ConsumeSink(system); subchannel->current_dma_segment = dma_state.dma_get + dma_state.dma_word_offset; @@ -679,38 +806,67 @@ void DmaPusher::CallMultiMethod(const u32* base_start, u32 num_methods) { } } -void DmaPusher::NoteDmaSemaphore(u32 method, u32 argument) { - using Engines::MaxwellDMA; - if (method >= MaxwellDMA::SemaphoreMethod && method < MaxwellDMA::SemaphoreMethod + 3) { - dma_semaphore[method - MaxwellDMA::SemaphoreMethod] = argument; +void DmaPusher::ShadowEngineMethod(Engines::EngineTypes type, u32 method, u32 argument) { + // As the engines' CallMethod and ConsumeSink store them. + if (type == Engines::EngineTypes::MaxwellDMA) { + if (method < dma_registers.size()) { + dma_registers[method] = argument; + } + } else if (type == Engines::EngineTypes::KeplerMemory) { + if (method < inline_to_memory_registers.size()) { + inline_to_memory_registers[method] = argument; + } + } +} + +void DmaPusher::NoteEngineWrites(Engines::EngineTypes type, u32 method) { + auto& split = VideoCommon::GpuSplit::Get(); + if (type == Engines::EngineTypes::KeplerMemory) { + if (method != KEPLERMEMORY_REG_INDEX(data)) { + return; + } + // Upload::State::ProcessData, of the upload KeplerMemory's exec started. + Engines::Upload::Registers upload; + std::memcpy(&upload, &inline_to_memory_registers[KEPLERMEMORY_REG_INDEX(upload)], + sizeof(upload)); + const bool linear = (inline_to_memory_registers[KEPLERMEMORY_REG_INDEX(exec)] & 1) != 0; + const auto range = Engines::Upload::UploadedRange(upload, linear); + split.NoteWrite(memory_manager, range.address, range.size); return; } - if (method != MaxwellDMA::LaunchDmaMethod) { + using Engines::MaxwellDMA; + if (type != Engines::EngineTypes::MaxwellDMA || method != MaxwellDMA::LaunchDmaMethod) { return; } - // As MaxwellDMA::ReleaseSemaphore writes it. - using SemaphoreType = MaxwellDMA::LaunchDMA::SemaphoreType; - const auto type = std::bit_cast(argument).semaphore_type.Value(); - if (type != SemaphoreType::RELEASE_ONE_WORD_SEMAPHORE && - type != SemaphoreType::RELEASE_FOUR_WORD_SEMAPHORE) { + const auto writes = MaxwellDMA::WritesOfLaunch(dma_registers); + split.NoteWrite(memory_manager, writes.destination.address, writes.destination.size); + if (writes.semaphore.size == 0) { return; } - const GPUVAddr address = MaxwellDMA::PackedGPUVAddr{dma_semaphore[0], dma_semaphore[1]}; - const auto device = memory_manager.GpuToCpuAddress(address); + // As MaxwellDMA::ReleaseSemaphore writes it. + split.NoteWrite(memory_manager, writes.semaphore.address, writes.semaphore.size); + const auto device = memory_manager.GpuToCpuAddress(writes.semaphore.address); if (!device) { return; } - auto& split = VideoCommon::GpuSplit::Get(); - split.NoteSemaphoreWrite(*device, dma_semaphore[2], true); - if (type == SemaphoreType::RELEASE_FOUR_WORD_SEMAPHORE) { + // Fence-flagged payload releases: they land once their fence ran, or as the back runs them + // where CounterReport keeps them synchronized. + const auto landing = VideoCommon::GpuSplit::FencedPayloadLanding(); + split.NoteSemaphoreWrite(*device, dma_registers[MaxwellDMA::SemaphoreMethod + 2], true, false, + landing); + VideoCommon::Nx2AcquireStats::NoteRelease(*device, + VideoCommon::Nx2AcquireStats::Releaser::Copy); + if (writes.semaphore.size > sizeof(u32)) { // The payload as 64 bits (a timestamp follows). - split.NoteSemaphoreWrite(*device + 4, 0, true); + split.NoteSemaphoreWrite(*device + 4, 0, true, false, landing); } } -void DmaPusher::RefreshSnapshot(CommandList& list, u64 now) { +void DmaPusher::RefreshSnapshot(CommandList& list, u64 now, size_t first_entry) { + VideoCommon::Nx2Profile::Scope profile_scope{VideoCommon::Nx2Profile::Zone::Resnapshot}; // TEST list.snapshot_suspensions = now; - for (size_t i = 0; i < list.command_lists.size() && i < list.submitted_offsets.size(); ++i) { + for (size_t i = first_entry; i < list.command_lists.size() && i < list.submitted_offsets.size(); + ++i) { const CommandListHeader& entry = list.command_lists[i]; const u32 offset = list.submitted_offsets[i]; if (offset + entry.size > list.submitted_words.size()) { @@ -718,6 +874,8 @@ void DmaPusher::RefreshSnapshot(CommandList& list, u64 now) { } memory_manager.ReadBlockUnsafe(entry.addr, list.submitted_words.data() + offset, entry.size * sizeof(u32)); + VideoCommon::Nx2Profile::Add(VideoCommon::Nx2Profile::Counter::ResnapshotWords, + entry.size); // TEST } list.deferred_entries.clear(); } @@ -751,14 +909,35 @@ bool DmaPusher::IsGpuWrittenCommands(GPUVAddr address, size_t size) { return false; } +void DmaPusher::CheckUnwaitedWriters(GPUVAddr address, size_t size) { + auto& split = VideoCommon::GpuSplit::Get(); + if (!split.Active() || !split.WritersPending()) { + return; + } + // The list did not wait for compute or copy work sent after the last sync. At the list's + // place, the back has run that work: did it make the list's pages GPU-written? + VideoCommon::Nx2DrainCheck::NoteListUnwaited(); + split.Defer( + [memory = &memory_manager, address, size] { + bool written = false; + for (GPUVAddr page = address & ~GPUVAddr{0xfff}; page < address + size; + page += 0x1000) { + const auto device = memory->GpuToCpuAddress(page); + written = written || (device && Tegra::Nx2IsGpuWritten(*device)); + } + VideoCommon::Nx2DrainCheck::NoteListChecked(written); + }, + VideoCommon::GpuSplit::Writes::Noted); +} + bool DmaPusher::ReadGpuWrittenCommands(GPUVAddr address, u32 count, bool* pending) { // The pages are marked as command list pages: from the next write on, the buffer cache copies // them to host memory right after the dispatch or copy that writes them and submits. The // front then waits only for the back to run that work and for the GPU to finish the copy, not // for everything queued and a cache flush (FlushGpuWrittenCommands: ~400 GPU round trips a - // second in a Mario Kart World race, during which the back had nothing to do). Pages - // FlushGpuWrittenCommands flushed whole and no GPU work wrote since are read from guest - // memory. + // second in a Mario Kart World race, during which the back had nothing to do). Lists on words + // no GPU work wrote since FlushGpuWrittenCommands flushed the page, or the host wrote them, + // are read from guest memory and wait for no copy (BufferCache::Nx2ReadListCopy). if (!Tegra::g_nx2_ampere.load(std::memory_order_relaxed) || count == 0 || rasterizer == nullptr) { return false; @@ -775,16 +954,22 @@ bool DmaPusher::ReadGpuWrittenCommands(GPUVAddr address, u32 count, bool* pendin if (!written) { return false; } + // As for lists on pages no GPU work wrote (Step): the dispatch or copy that rewrites the + // list is ordered before the fetch only by a host-level wait between them, so the front waits + // for the work before the last one, not for compute and copy work sent after it. auto& split = VideoCommon::GpuSplit::Get(); - // Resuming after the copy's tick: the back ran the list's writer before the suspension + // Resuming after the copy's tick: the back ran the list's writers before the suspension // (this wait ran then), and waiting again would wait for the other channels' work queued // meanwhile too. const bool resumed = std::exchange(resumed_list_read, false); if (split.Active() && !resumed) { - split.WaitForWriters(); + split.WaitForSyncedWriters(VideoCommon::DrainSite::ListCopies); } bool past_sync = !split.Active(); using GpuWrittenList = VideoCore::RasterizerInterface::GpuWrittenList; + if (Nx2ListStats::Any()) [[unlikely]] { + Nx2ListStats::BeginList(count); + } command_headers.resize_destructive(count); u8* const out = reinterpret_cast(command_headers.data()); for (GPUVAddr chunk = address; chunk < end;) { @@ -796,9 +981,13 @@ bool DmaPusher::ReadGpuWrittenCommands(GPUVAddr address, u32 count, bool* pendin } GpuWrittenList where = GpuWrittenList::Synced; if (Tegra::Nx2IsGpuWritten(*device)) { + const bool stats = Nx2ListStats::g_enabled && split.Active(); + const u64 backlog = stats ? split.Backlog() : 0; + const u64 unpublished = stats ? split.Unpublished() : 0; u64 tick = 0; u64* const pending_tick = pending != nullptr ? &tick : nullptr; where = rasterizer->ReadGpuWrittenCommandList(*device, part, past_sync, pending_tick); + const bool after_sync = where == GpuWrittenList::NeedsSync; if (where == GpuWrittenList::NeedsSync) { const auto wait_start = Nx2CmdPageDiag::g_enabled ? std::chrono::steady_clock::now() @@ -826,10 +1015,13 @@ bool DmaPusher::ReadGpuWrittenCommands(GPUVAddr address, u32 count, bool* pendin diag.reads_synced.fetch_add(1, std::memory_order_relaxed); if (split.Active() && Nx2CmdPageDiag::IsDrawPage(page)) { // The back, at this list's place: did work sent before it write the page? - split.Defer([page, begin = static_cast(*device - page), - end = static_cast(*device - page + part.size())] { - Nx2CmdPageDiag::CheckMissedWrite(page, begin, end); - }, VideoCommon::GpuSplit::FastDrawNeutral); + split.Defer( + [page, begin = static_cast(*device - page), + end = static_cast(*device - page + part.size())] { + Nx2CmdPageDiag::CheckMissedWrite(page, begin, end); + }, + VideoCommon::GpuSplit::Writes::Anywhere, + VideoCommon::GpuSplit::FastDrawNeutral); } } else { diag.reads_unknown.fetch_add(1, std::memory_order_relaxed); @@ -838,6 +1030,10 @@ bool DmaPusher::ReadGpuWrittenCommands(GPUVAddr address, u32 count, bool* pendin } } } + if (Nx2ListStats::Any()) [[unlikely]] { + NoteListRead(memory_manager, channel_state.bind_id, chunk, *device, part, where, + after_sync, backlog, unpublished); + } } if (where == GpuWrittenList::Unknown) { return false; @@ -847,9 +1043,148 @@ bool DmaPusher::ReadGpuWrittenCommands(GPUVAddr address, u32 count, bool* pendin } chunk = chunk_end; } + if (Nx2ListStats::g_verify) [[unlikely]] { + VerifyGpuWrittenCommands(address, count); + } return true; } +bool DmaPusher::TakeGpuWrittenListWords(const CommandListHeader& header) { + // Ocarina of Time's compute culling writes the group counts of a later dispatch as the + // parameters of compute macro 0xe74, which only stores them in the data RAM for 0xe7a and + // 0xe76 to upload and launch with. The front need not wait for the culling to run and its + // copy to come back: the macros run on placeholders the host GPU's copy of the words takes + // the place of (Nx2ListIndirect, KeplerCompute::FrontTakeListWords). + auto& split = VideoCommon::GpuSplit::Get(); + using Kept = Nx2ListIndirect::Stats::Kept; // TEST + if (Nx2ListIndirect::Stats::Enabled()) [[unlikely]] { + Nx2ListIndirect::Stats::MaybeLog(); + } + if (!Nx2ListIndirect::g_enabled || rasterizer == nullptr || command_resume_offset != 0 || + !split.Active() || !split.FrontMacros()) { + Nx2ListIndirect::Stats::NoteKept(Kept::Off); + return false; + } + const u32 count = static_cast(header.size); + // All of a macro call's parameters, from its first: its header ended the entry before. + if (count == 0 || count > Nx2ListIndirect::SlotWords || dma_state.method_count != count || + dma_state.method < MacroRegistersStart || (dma_state.method & 1) != 0 || + subchannels[dma_state.subchannel] == nullptr || + subchannel_type[dma_state.subchannel] != Engines::EngineTypes::KeplerCompute) { + Nx2ListIndirect::Stats::NoteKept(Kept::NotCall); + return false; + } + auto* const compute = static_cast(subchannels[dma_state.subchannel]); + if (!compute->FrontRunsMacros() || !rasterizer->Nx2ListSlotsSupported()) { + Nx2ListIndirect::Stats::NoteKept(Kept::Off); + return false; + } + // On one device page, as the back finds it: its mappings are the front's once it ran the + // last change sent. + const u64 bytes = u64{count} * sizeof(u32); + const auto device = memory_manager.GpuToCpuAddress(dma_state.dma_get); + const auto last = memory_manager.GpuToCpuAddress(dma_state.dma_get + bytes - 1); + if (!device || !last || *last != *device + bytes - 1 || !split.Ran(split.LastMappingChange())) { + Nx2ListIndirect::Stats::NoteKept(Kept::Placement); + return false; + } + const std::optional slot = rasterizer->Nx2AcquireListSlot(); + if (!slot) { + Nx2ListIndirect::Stats::NoteKept(Kept::NoSlot); + return false; + } + if (!compute->FrontTakeListWords(split, dma_state.method, count, *slot)) { + rasterizer->Nx2ReleaseListSlot(*slot); // Nothing sent with it + Nx2ListIndirect::Stats::NoteKept(Kept::Refused); + return false; + } + Nx2ListIndirect::Stats::NoteTaken(dma_state.method, count); + // TEST (EDEN_NX2_LIST_INDIRECT_CHECK): the words as the old path reads them (waiting for + // their writers), for the back to compare what the host GPU uses of the slot with. + const bool check = Nx2ListIndirect::Stats::g_check; + std::array expected{}; + if (check) [[unlikely]] { + if (ReadGpuWrittenCommands(dma_state.dma_get, count)) { + for (u32 k = 0; k < count; ++k) { + expected[k] = command_headers[k].argument; + } + } else { + FlushGpuWrittenCommands(dma_state.dma_get, bytes); + memory_manager.ReadBlockUnsafe(dma_state.dma_get, expected.data(), bytes); + } + } + // At the list's place: after the work sent before it, the GPU fetched the list there. + split.Defer( + [rasterizer_ = rasterizer, device_ = *device, count, slot_ = *slot, check, expected] { + rasterizer_->Nx2SnapshotListWords(device_, count, slot_); + if (check) { + rasterizer_->Nx2CheckListSlot(slot_, std::span{expected}.first(count)); + } + }, + VideoCommon::GpuSplit::Writes::Noted); + command_headers.resize_destructive(count); + for (u32 k = 0; k < count; ++k) { + command_headers[k].argument = Nx2ListIndirect::Placeholder(*slot, k); + } + return true; +} + +void DmaPusher::VerifyGpuWrittenCommands(GPUVAddr address, u32 count) { + // The list again as the front read lists before it tracked the words GPU work wrote: after + // every dispatch and copy sent so far, from the pages' whole copies, or drained and flushed. + using GpuWrittenList = VideoCore::RasterizerInterface::GpuWrittenList; + const size_t bytes = size_t{count} * sizeof(u32); + std::vector parsed(count); + std::memcpy(parsed.data(), command_headers.data(), bytes); + auto& split = VideoCommon::GpuSplit::Get(); + if (split.Active()) { + split.WaitForWriters(); + } + bool past_sync = !split.Active(); + std::vector reference(count); + bool reference_drained = false; + { + const Nx2ListStats::ReferenceScope reference_read; + u8* const out = reinterpret_cast(reference.data()); + const GPUVAddr end = address + bytes; + for (GPUVAddr chunk = address; chunk < end && !reference_drained;) { + const GPUVAddr chunk_end = (std::min)(end, (chunk & ~GPUVAddr{0xfff}) + 0x1000); + const std::span part{out + (chunk - address), + static_cast(chunk_end - chunk)}; + const auto device = memory_manager.GpuToCpuAddress(chunk); + GpuWrittenList where = device ? GpuWrittenList::Synced : GpuWrittenList::Unknown; + if (device && Tegra::Nx2IsGpuWritten(*device)) { + where = rasterizer->ReadGpuWrittenCommandList(*device, part, past_sync); + if (where == GpuWrittenList::NeedsSync) { + split.WaitForSync(); + past_sync = true; + where = rasterizer->ReadGpuWrittenCommandList(*device, part, true); + } + } + if (where == GpuWrittenList::Synced) { + memory_manager.ReadBlockUnsafe(chunk, part.data(), part.size()); + } + reference_drained = where == GpuWrittenList::Unknown; + chunk = chunk_end; + } + } + const auto read_drained = [&](std::vector& words) { + FlushGpuWrittenCommands(address, bytes); + memory_manager.ReadBlockUnsafe(address, words.data(), bytes); + }; + if (reference_drained) { + read_drained(reference); + } + std::vector drained; + if (!reference_drained && reference != parsed) { + drained.resize(count); + read_drained(drained); + } + Nx2ListStats::NoteVerify(address, parsed, reference, drained, reference_drained); + // The front parses the words read last. + std::memcpy(command_headers.data(), drained.empty() ? reference.data() : drained.data(), bytes); +} + bool DmaPusher::FlushGpuWrittenCommands(GPUVAddr address, size_t size) { if (!g_nx2_ampere.load(std::memory_order_relaxed) || size == 0) { return false; diff --git a/src/video_core/dma_pusher.h b/src/video_core/dma_pusher.h index dbf8d0355d..e8177fbdb7 100644 --- a/src/video_core/dma_pusher.h +++ b/src/video_core/dma_pusher.h @@ -182,6 +182,7 @@ class DmaPusher final { void BindSubchannel(Engines::EngineInterface* engine, u32 subchannel_id, Engines::EngineTypes engine_type) { subchannels[subchannel_id] = engine; subchannel_type[subchannel_id] = engine_type; + engine->bound_type = engine_type; } void BindRasterizer(VideoCore::RasterizerInterface* rasterizer); @@ -216,11 +217,19 @@ class DmaPusher final { void RecordParsedEntry(const CommandListHeader& header, std::span commands); /// Whether the entry lies on device pages the GPU has written (NX2). bool IsGpuWrittenCommands(GPUVAddr address, size_t size); - /// A copy engine method sent to the split's back: a launch that releases its semaphore tells - /// the front what it writes (GpuSplit::NoteSemaphoreWrite). - void NoteDmaSemaphore(u32 method, u32 argument); - /// Copies the list's words from memory again, as of suspension count `now`. - void RefreshSnapshot(CommandList& list, u64 now); + /// EDEN_NX2_DRAIN_CHECK: checks an entry on pages no GPU work had written against the compute + /// and copy work it did not wait for (Nx2DrainCheck). + void CheckUnwaitedWriters(GPUVAddr address, size_t size); + /// Keeps the front's copy of the copy and inline-to-memory engines' registers, with the split + /// and without. + void ShadowEngineMethod(Engines::EngineTypes type, u32 method, u32 argument); + /// An engine method was just sent to the split's back, as it is: places what a copy launch or + /// an inline-to-memory upload writes (GpuSplit::NoteWrite), and tells what a launch releases + /// (GpuSplit::NoteSemaphoreWrite). + void NoteEngineWrites(Engines::EngineTypes type, u32 method); + /// Copies the list's words from memory again, as of suspension count `now`, from entry + /// `first_entry` on (the entries before it ran already). + void RefreshSnapshot(CommandList& list, u64 now, size_t first_entry = 0); /// Copies the list's deferred entries (CommandList::deferred_entries) from memory. void RefreshDeferredEntries(CommandList& list); @@ -238,6 +247,14 @@ class DmaPusher final { /// flushes and reads guest memory. With `pending`, a copy the GPU has not finished sets /// `*pending` and pending_list_tick instead of waiting (the result is false then). bool ReadGpuWrittenCommands(GPUVAddr address, u32 count, bool* pending = nullptr); + /// A GPU-written list that is all the parameters of the compute macro call about to start, + /// on the split with macros on its front: if the call only moves the words where the back can + /// take them over (Nx2ListIndirect), puts their placeholders in command_headers instead of + /// reading them. The back copies the words at the list's place on the host GPU. + bool TakeGpuWrittenListWords(const CommandListHeader& header); + /// EDEN_NX2_LIST_VERIFY: reads the list ReadGpuWrittenCommands just read again as before + /// (Nx2ListStats) and leaves the words read last in command_headers. + void VerifyGpuWrittenCommands(GPUVAddr address, u32 count); /// Brings command list words the GPU wrote (still only in host buffers) to guest memory. bool FlushGpuWrittenCommands(GPUVAddr address, size_t size); @@ -303,8 +320,13 @@ class DmaPusher final { std::array subchannels{}; std::array subchannel_type; - /// The copy engine's semaphore address (high, low) and payload as sent (NoteDmaSemaphore). - std::array dma_semaphore{}; + /// The registers of the channel's copy engine (MaxwellDMA::RegisterCount) and its + /// inline-to-memory engine (KeplerMemory::Regs::NUM_REGS), as the methods sent left them + /// (ShadowEngineMethod): where the back's copies and uploads write. + static constexpr size_t DmaRegisterCount = 0x800; + static constexpr size_t InlineToMemoryRegisterCount = 0x7f; + std::array dma_registers{}; + std::array inline_to_memory_registers{}; Engines::Puller puller; std::mutex sync_mutex; diff --git a/src/video_core/engines/engine_interface.h b/src/video_core/engines/engine_interface.h index 7f1a603c20..21f06bf8e2 100644 --- a/src/video_core/engines/engine_interface.h +++ b/src/video_core/engines/engine_interface.h @@ -8,6 +8,7 @@ #include #include +#include #include #include "common/common_types.h" @@ -50,9 +51,13 @@ class EngineInterface { std::vector> method_sink{}; GPUVAddr current_dma_segment; bool current_dirty{}; + /// The class DmaPusher::BindSubchannel bound the engine as, if it did: logs name engines by + /// it, as Lichii builds without RTTI (/GR-, -fno-rtti). + std::optional bound_type; /// Fast draw (nx2_fast_draw.h): the engine reports its own changes (Maxwell3D); a method of any other /// engine ends draw reuse. bool reports_fast_draw_breaks{}; + protected: virtual void ConsumeSinkImpl(Core::System& system) { for (auto [method, value] : method_sink) { diff --git a/src/video_core/engines/engine_upload.cpp b/src/video_core/engines/engine_upload.cpp index 0ffdae8036..e4799db593 100644 --- a/src/video_core/engines/engine_upload.cpp +++ b/src/video_core/engines/engine_upload.cpp @@ -16,6 +16,32 @@ namespace Tegra::Engines::Upload { +namespace { + +/// The power of two bytes a swizzled upload moves per element (ProcessData). +u32 SwizzleBppShift(const Registers& regs) { + return Common::FoldRight( + 4U, [](u32 x, u32 y) { return (std::min)(x, static_cast(std::countr_zero(y))); }, + regs.dest.width, regs.line_length_in, regs.dest.x, static_cast(regs.dest.Address())); +} + +} // Anonymous namespace + +WrittenRange UploadedRange(const Registers& regs, bool is_linear) { + const GPUVAddr address = regs.dest.Address(); + if (is_linear) { + if (regs.line_count == 0 || regs.line_length_in == 0) { + return {}; + } + // The last line ends last, whatever the pitch. + return {address, u64{regs.dest.pitch} * (regs.line_count - 1) + regs.line_length_in}; + } + const u32 bpp_shift = SwizzleBppShift(regs); + return {address, Tegra::Texture::CalculateSize( + true, 1U << bpp_shift, regs.dest.width >> bpp_shift, regs.dest.height, + regs.dest.depth, regs.dest.BlockHeight(), regs.dest.BlockDepth())}; +} + State::State(MemoryManager& memory_manager_, Registers& regs_) : regs{regs_}, memory_manager{memory_manager_} {} @@ -77,15 +103,11 @@ void State::ProcessData(std::span read_buffer) { rasterizer->AccelerateInlineToMemory(dest_line, regs.line_length_in, buffer); } } else { - u32 width = regs.dest.width; - u32 x_elements = regs.line_length_in; - u32 x_offset = regs.dest.x; - const u32 bpp_shift = Common::FoldRight( - 4U, [](u32 x, u32 y) { return (std::min)(x, static_cast(std::countr_zero(y))); }, - width, x_elements, x_offset, static_cast(address)); - width >>= bpp_shift; - x_elements >>= bpp_shift; - x_offset >>= bpp_shift; + // As UploadedRange places it. + const u32 bpp_shift = SwizzleBppShift(regs); + const u32 width = regs.dest.width >> bpp_shift; + const u32 x_elements = regs.line_length_in >> bpp_shift; + const u32 x_offset = regs.dest.x >> bpp_shift; const u32 bytes_per_pixel = 1U << bpp_shift; const std::size_t dst_size = Tegra::Texture::CalculateSize( true, bytes_per_pixel, width, regs.dest.height, regs.dest.depth, diff --git a/src/video_core/engines/engine_upload.h b/src/video_core/engines/engine_upload.h index 55a82f938b..8169ce8cba 100644 --- a/src/video_core/engines/engine_upload.h +++ b/src/video_core/engines/engine_upload.h @@ -57,6 +57,17 @@ struct Registers { } dest; }; +/// Guest memory [address, address + size) an upload writes (size 0: none). +struct WrittenRange { + GPUVAddr address{}; + u64 size{}; +}; + +/// Where State::ProcessData writes an upload with registers `regs` (linear: lines of +/// line_length_in bytes, pitch apart; else swizzled into the destination surface): what the split +/// GPU thread's front notes for the back to write running the upload's data methods. +[[nodiscard]] WrittenRange UploadedRange(const Registers& regs, bool is_linear); + class State { public: explicit State(MemoryManager& memory_manager_, Registers& regs_); diff --git a/src/video_core/engines/kepler_compute.cpp b/src/video_core/engines/kepler_compute.cpp index 9069828f46..caa9c8177d 100644 --- a/src/video_core/engines/kepler_compute.cpp +++ b/src/video_core/engines/kepler_compute.cpp @@ -18,13 +18,18 @@ #include "common/assert.h" #include "common/dkb_frame_trace.h" #include "common/logging.h" +#include "common/scope_exit.h" #include "common/settings.h" #include "shader_recompiler/exception.h" #include "core/core.h" #include "video_core/gpu_split.h" #include "video_core/fire_emblem_upscale.h" +#include "video_core/front_qmd.h" #include "video_core/nx2_profile.h" #include "video_core/nx2.h" +#include "video_core/nx2_drain_check.h" +#include "video_core/nx2_list_indirect.h" +#include "video_core/nx2_list_indirect_stats.h" #include "video_core/engines/fermi_2d.h" #include "video_core/engines/kepler_compute.h" #include "video_core/engines/native_upscale.h" @@ -78,7 +83,11 @@ KeplerCompute::KeplerCompute(MemoryManager& memory_manager_) } } -KeplerCompute::~KeplerCompute() = default; +KeplerCompute::~KeplerCompute() { + if (native_data_ram != nullptr) { + Nx2ListIndirect::ForgetChannel(native_data_ram); + } +} void KeplerCompute::BindRasterizer(VideoCore::RasterizerInterface* rasterizer_) { rasterizer = rasterizer_; @@ -307,6 +316,12 @@ bool KeplerCompute::ProcessNativeMethod(Core::System& system, u32 method, u32 ar } rasterizer->AccelerateInlineToMemory(address, 256, std::span{reinterpret_cast(native_registers.data() + 0xc8), 256}); + if (const auto device = memory_manager.GpuToCpuAddress(address)) { + // The slots the grid's list word placeholders name stay for its launches. + rasterizer->Nx2NoteInlineQmdGrid(*device, native_registers[0xc8 + 12], + native_registers[0xc8 + 13], + native_registers[0xc8 + 14]); + } if ((control & (1U << 24)) || !(native_registers[0xcc] & (1U << 12))) { const u32 previous = native_registers[0xad]; native_registers[0xad] = static_cast(address >> 8); @@ -348,6 +363,11 @@ bool KeplerCompute::ProcessNativeMethod(Core::System& system, u32 method, u32 ar [&](u64 address, NativeQmdScheduler::Descriptor& descriptor) { if (!memory_manager.IsFullyMappedRange(address, sizeof(descriptor))) return false; memory_manager.ReadBlockUnsafe(address, descriptor.data(), sizeof(descriptor)); + if (VideoCommon::Nx2DrainCheck::g_enabled) [[unlikely]] { + // TEST: what the front's QMD view compares the descriptors it read with. + VideoCommon::Nx2DrainCheck::NoteQmdRead(address, descriptor, + VideoCommon::GpuSplit::OnBackThread()); + } if (static u32 logged{}; (descriptor[53] & (1U << 23)) && logged < 8) { ++logged; LOG_INFO(HW_GPU, "Native QMD {:#x} releases: {:08x} | {:08x} | {:08x}", address, @@ -368,18 +388,30 @@ bool KeplerCompute::ProcessNativeMethod(Core::System& system, u32 method, u32 ar [&](u64 address, u32 payload, u32 flags) { const bool four_words = (flags >> 30) == 0; if (!memory_manager.IsFullyMappedRange(address, four_words ? 16 : sizeof(payload))) return false; + // TEST (EDEN_NX2_DRAIN_CHECK): what the front's QMD view compares its notes with. + const auto checked = [this, address](u32 offset, u32 value) { + if (VideoCommon::Nx2DrainCheck::g_enabled) [[unlikely]] { + if (const auto device = memory_manager.GpuToCpuAddress(address)) { + VideoCommon::Nx2DrainCheck::NoteQmdRelease( + *device + offset, value, VideoCommon::GpuSplit::OnBackThread()); + } + } + }; if (flags & (1U << 26)) { - rasterizer->SignalFence([this, address, payload, flags] { + rasterizer->SignalFence([this, address, payload, flags, checked] { const auto combine = [&](u32 current) { - return ReduceSemaphore((flags >> 20) & 7, (flags >> 24) & 1, current, payload); + return ReduceSemaphore((flags >> 20) & 7, (flags >> 24) & 1, current, + payload); }; const u32 guest = memory_manager.Read(address); const u32 value = combine(guest); memory_manager.Write(address, value); + checked(0, value); if (const auto device = memory_manager.GpuToCpuAddress(address)) { Tegra::Nx2SemaphoreWaiters::Get().OnWrite( *device, Tegra::Nx2MirrorShadow::Get().NoteCombinedRelease( - *device, address, guest, value, combine)); + *device, address, guest, value, combine), + Tegra::Nx2SemaphoreWaiters::Source::QmdReduction); } static const u64 watch = [] { const char* text = std::getenv("EDEN_NX2_SEM_WATCH"); @@ -395,19 +427,25 @@ bool KeplerCompute::ProcessNativeMethod(Core::System& system, u32 method, u32 ar rasterizer->Query(address, VideoCommon::QueryType::Payload, VideoCommon::QueryPropertiesFlags::IsAFence | VideoCommon::QueryPropertiesFlags::HasTimeout, payload, 0); + checked(0, payload); + checked(4, 0); } else if (flags & (1U << 19)) { // NVN2 tracks QMD completion with a max release (bit 19) into a word other // releases also advance: a plain write would move it backwards and leave // GEQ acquires waiting forever. - rasterizer->SignalFence([this, address, payload] { - const auto combine = [payload](u32 current) { return (std::max)(current, payload); }; + rasterizer->SignalFence([this, address, payload, checked] { + const auto combine = [payload](u32 current) { + return (std::max)(current, payload); + }; const u32 guest = memory_manager.Read(address); const u32 value = combine(guest); memory_manager.Write(address, value); + checked(0, value); if (const auto device = memory_manager.GpuToCpuAddress(address)) { Tegra::Nx2SemaphoreWaiters::Get().OnWrite( *device, Tegra::Nx2MirrorShadow::Get().NoteCombinedRelease( - *device, address, guest, value, combine)); + *device, address, guest, value, combine), + Tegra::Nx2SemaphoreWaiters::Source::QmdMax); } }); } else { @@ -420,9 +458,14 @@ bool KeplerCompute::ProcessNativeMethod(Core::System& system, u32 method, u32 ar } rasterizer->Query(address, VideoCommon::QueryType::Payload, VideoCommon::QueryPropertiesFlags::IsAFence, payload, 0); + checked(0, payload); } return true; }, [&](const auto& descriptor) { return DispatchNativeCompute(descriptor); }); + if (VideoCommon::Nx2DrainCheck::g_enabled) [[unlikely]] { + // TEST: where this signal's reads end, for the front's QMD view check. + VideoCommon::Nx2DrainCheck::NoteQmdSignalEnd(VideoCommon::GpuSplit::OnBackThread()); + } if (error.empty()) { LOG_DEBUG(HW_GPU, "Native QMD control action={:#x}, QMD={:#x}", argument, qmd_address); return true; @@ -542,7 +585,38 @@ void KeplerCompute::CallMultiMethod(Core::System& system, u32 method, const u32* } } -bool KeplerCompute::DispatchNativeCompute(const NativeQmdScheduler::Descriptor& qmd) { +bool KeplerCompute::DispatchNativeCompute(const NativeQmdScheduler::Descriptor& descriptor) { + // A grid of GPU-written list words (Nx2ListIndirect): one slot's three words, which the host + // GPU dispatches from, or placeholders of several, whose words the back reads. + NativeQmdScheduler::Descriptor resolved; + const bool listed = Nx2ListIndirect::IsPlaceholder(descriptor[12]) || + Nx2ListIndirect::IsPlaceholder(descriptor[13]) || + Nx2ListIndirect::IsPlaceholder(descriptor[14]); + if (listed) [[unlikely]] { + indirect_list_slot = + Nx2ListIndirect::GridSlot(descriptor[12], descriptor[13], descriptor[14]); + if (!indirect_list_slot) { + resolved = descriptor; + for (const u32 word : {12U, 13U, 14U}) { + if (Nx2ListIndirect::IsPlaceholder(resolved[word])) { + const u32 slot = Nx2ListIndirect::PlaceholderSlot(resolved[word]); + std::array words{}; + rasterizer->Nx2ReadListSlots(std::span{&slot, 1}, words); + Nx2ListIndirect::Stats::Count(Nx2ListIndirect::Stats::Get().grids_read); + resolved[word] = words[Nx2ListIndirect::PlaceholderWord(resolved[word])]; + } + } + // An empty grid dispatches nothing (NativeQmdScheduler::Signal does not call this). + if (!(resolved[12] && (resolved[13] & 0xffff) && (resolved[14] & 0xffff))) { + return true; + } + } + } + SCOPE_EXIT { + indirect_list_slot.reset(); + }; + const NativeQmdScheduler::Descriptor& qmd = + listed && !indirect_list_slot ? resolved : descriptor; launch_description = {}; launch_description.grid_dim_x.Assign(qmd[12]); launch_description.grid_dim_y.Assign(qmd[13] & 0xffff); @@ -609,7 +683,7 @@ bool KeplerCompute::DispatchNativeCompute(const NativeQmdScheduler::Descriptor& return from_width >= 64 && to_width > from_width && to_width <= from_width * 4 && (cross_a > cross_b ? cross_a - cross_b : cross_b - cross_a) * 32 <= cross_a; }; - const auto [reads, stores] = NativeImageHandles(regs.code_loc.Address()); + const auto& [reads, stores] = NativeImageHandles(regs.code_loc.Address()); std::optional replacement_input; std::optional replacement_output; if (native_upscale_input && network_skip_remaining > 0 && !stores.empty()) { @@ -1152,7 +1226,8 @@ std::atomic g_nx2_trace_draws_after_upscale{}; // The cbuf0 byte offsets of the texture handles a native program reads (TEX, TLD, TLD4, TXD) and // the surface handles it stores to (SUST.P), from its code up to the first unconditional EXIT. -std::pair, std::vector> KeplerCompute::NativeImageHandles(GPUVAddr program) { +const std::pair, std::vector>& KeplerCompute::NativeImageHandles( + GPUVAddr program) { static std::unordered_map, std::vector>> cache; if (const auto it = cache.find(program); it != cache.end()) { return it->second; @@ -1190,7 +1265,7 @@ std::pair, std::vector> KeplerCompute::NativeImageHandles( void KeplerCompute::RememberNativeUpscaleInput() { native_upscale_input.reset(); - const auto [reads, stores] = NativeImageHandles(regs.code_loc.Address()); + const auto& [reads, stores] = NativeImageHandles(regs.code_loc.Address()); // An image-fed matrix pass writes its intermediate results to buffers. Retain its colour // input: the final pass's sampled image is another intermediate that a skipped network never // produced (Ocarina's movie network reads its colour at +0x58). @@ -1424,9 +1499,157 @@ Texture::TSCEntry KeplerCompute::GetTSCEntry(u32 tsc_index) const { return tsc_entry; } +namespace { + +// The QMD scheduling of every channel, run on the front as the back will run it. +VideoCommon::FrontQmdView& GetFrontQmd() { + return VideoCommon::FrontQmdView::Get(); +} + +// An engine's GPU address space as its back reads QMDs and their release words (FrontQmdView). +struct FrontQmdMemory { + const MemoryManager& memory; + + [[nodiscard]] bool IsMapped(GPUVAddr address, u64 size) const { + return memory.IsFullyMappedRange(address, size); + } + void Read(GPUVAddr address, void* data, u64 size) const { + memory.ReadBlockUnsafe(address, data, size); + } + [[nodiscard]] u32 ReadWord(GPUVAddr address) const { + return memory.Read(address); + } + [[nodiscard]] bool InSyncpointShim(GPUVAddr address, u64 size) const { + return memory.OverlapsSyncpointShim(address, size); + } + [[nodiscard]] std::optional Device(GPUVAddr address) const { + return memory.GpuToCpuAddress(address); + } + [[nodiscard]] bool GpuWritten(u64 device, u64 size) const { + return Tegra::Nx2MayBeGpuWritten(device, size); + } + [[nodiscard]] u64 WritesTarget(const VideoCommon::GpuSplit& split, GPUVAddr address, + u64 size) const { + return split.MemoryWritesTarget(memory, address, size); + } + void NoteWrite(VideoCommon::GpuSplit& split, GPUVAddr address, u64 size) const { + split.NoteWrite(memory, address, size); + } + void NoteWriteAt(VideoCommon::GpuSplit& split, GPUVAddr address, u64 size, u64 position) const { + split.NoteWriteAt(memory, address, size, position); + } + [[nodiscard]] const void* Id() const { + return &memory; + } +}; + +// The QMD methods: PCAS (0xae-0xb0) and the inline QMD's last word (0x107). +[[nodiscard]] bool IsQmdMethod(u32 method) { + return method == 0xae || method == 0xaf || method == 0xb0 || method == 0x107; +} + +// Methods whose notes the front takes from guest memory as the back finds it running them: the +// QMD methods (descriptors, and the words maxima and reductions combine with) and the compute +// report (reductions). The front holds them unpublished until it noted them. +[[nodiscard]] bool ValuedFromMemory(u32 method) { + return IsQmdMethod(method) || method == 0x6c3; +} + +// Whether a PCAS method signals the QMD scheduler, as ProcessNativeMethod filters them. +[[nodiscard]] bool SignalsQmd(u32 method, u32 argument) { + if (method == 0xaf) { + return (argument & 2) != 0; + } + return method == 0xb0 && (argument & 15) != 0 && (argument & 15) != 1; +} + +// What the back writes to guest memory running `method` (GpuSplit::Writes), as +// ProcessNativeMethod runs it: below the macro methods, nothing for register writes, the macro +// code and bindings (the front's) and the methods EDEN_NATIVE_DRAW_DIAGNOSTICS traces, and what +// FrontNoteWrites notes for reports, inline QMDs, QMD launches and inline uploads. +[[nodiscard]] VideoCommon::GpuSplit::Writes FrontWrites(u32 method) { + using Writes = VideoCommon::GpuSplit::Writes; + return method < 0xe00 ? Writes::Noted : Writes::Unplaced; +} + +// TEST (EDEN_NX2_DRAIN_CHECK): whether QMD maximum and reduction releases land as the back runs +// them (FenceManager::SignalFence runs its operation at once), for the QMD view check. +[[nodiscard]] bool QmdReleasesLandAtOnce() { + return Settings::IsGPUFenceBehaviorDefault() + ? !Settings::IsGPULevelHigh() + : !Settings::IsGPUFenceBehaviorBalanced() && !Settings::IsGPUFenceBehaviorAccurate(); +} + +using QmdTakeOver = VideoCommon::Nx2AcquireStats::QmdTakeOver; // TEST + +// A compute macro call as Nx2ListIndirect::ComputeFront sees it (its Context). +template +struct ListCallContext { + std::span code; + u64 generation; + u32 start; + std::vector& params; + std::span data_ram; + std::span registers; + Nx2ListIndirect::UploadState upload; + VideoCommon::GpuSplit& split; + const MemoryManager& memory; + VideoCore::RasterizerInterface* rasterizer; + EngineInterface* engine; + std::array& front_registers; + std::array& engine_registers; ///< Set directly once the split stopped + const EmitFunc& emit; + + [[nodiscard]] std::optional DeviceWord(u64 gpu_address) const { + if (!split.Ran(split.LastMappingChange())) { + return std::nullopt; // The back may find other mappings + } + const auto first = memory.GpuToCpuAddress(gpu_address); + const auto last = memory.GpuToCpuAddress(gpu_address + 3); + if (!first || !last || *last != *first + 3) { + return std::nullopt; + } + return u64{*first}; + } + void Emit(u32 method, u32 value) const { + emit(method, value); + } + void SendOwed(u64 device, u32 value) const { + split.Defer([rasterizer_ = rasterizer, device, + value] { rasterizer_->Nx2NoteOwedListWord(device, value); }, + VideoCommon::GpuSplit::Writes::Noted); + } + void Release(u32 slot) const { + Nx2ListIndirect::ReleaseSlot(split, rasterizer, slot); + } + void ReadSlots(std::span slots, std::span words) const { + const u64 read_start = + Nx2ListIndirect::Stats::Enabled() ? Nx2ListIndirect::Stats::NowNs() : 0; // TEST + Nx2ListIndirect::ReadSlots(split, rasterizer, slots, words); + if (Nx2ListIndirect::Stats::Enabled()) [[unlikely]] { + Nx2ListIndirect::Stats::NoteRead(slots.size(), + Nx2ListIndirect::Stats::NowNs() - read_start); + } + } + void SetRegister(u32 method, u32 value) const { + front_registers[method] = value; + if (method == Nx2ListIndirect::UploadData) { + return; // The upload's data port: only macros read it back, from the front's copy + } + if (split.Active()) { + split.MacroWrite(engine, method, value, VideoCommon::GpuSplit::Writes::Noted); + } else { + engine_registers[method] = value; + } + } +}; + +} // Anonymous namespace + // Split GPU thread with macros on its front half (gpu_split.h, EDEN_NX2_SPLIT=2): the front // runs compute macros against its own copy of the native registers (which hold the last value -// written to each, on the back too) and loads and binds the compute macro code itself. +// written to each, on the back too) and loads and binds the compute macro code itself. It also +// runs the QMD scheduling ahead of the back (FrontQmdView), to tell what the launches release. struct KeplerCompute::FrontState { std::array registers{}; std::optional executing_macro; @@ -1434,6 +1657,8 @@ struct KeplerCompute::FrontState { bool compute_uploaded{}; u32 bind_pointer{}; MacroEngine* macro{}; ///< Without a native context: the one macro engine + /// The inline upload the back runs, where a word of it lands (Nx2ListIndirect). + Nx2ListIndirect::UploadState upload; }; KeplerCompute::FrontState& KeplerCompute::Front(VideoCommon::GpuSplit& split) { @@ -1450,12 +1675,38 @@ KeplerCompute::FrontState& KeplerCompute::Front(VideoCommon::GpuSplit& split) { front->bind_pointer = native_bind_pointer; front->macro = native_macro; front_macros = true; + // A new session's or a restarted split's: the QMDs the back scheduled meanwhile, in the + // table the engines share. The back is idle, its table the current one. + GetFrontQmd().TakeOver(native_qmd, QmdTakeOver::FrontStart); } return *front; } // As Maxwell3D::ReleaseFront: the split stopped, the engine runs its macros again. void KeplerCompute::ReleaseFront() { + if (list_front && native_data_ram != nullptr) { + // They run on the GPU-written list words themselves from now on (Nx2ListIndirect). + const auto no_emit = [](u32, u32) { return false; }; + ListCallContext context{{}, + 0, + 0, + front->parameters, + std::span{*native_data_ram}, + front->registers, + front->upload, + VideoCommon::GpuSplit::Get(), + memory_manager, + rasterizer, + this, + front->registers, + native_registers, + no_emit}; + auto& channel = Nx2ListIndirect::GetChannel(native_data_ram); + if (!channel.words.empty() || !list_front->Registers().empty()) { + Nx2ListIndirect::Stats::NoteConcretized(Nx2ListIndirect::Reason::Stop); // TEST + } + list_front->ConcretizeAll(context, channel); + } native_executing_macro = front->executing_macro; native_parameters = std::move(front->parameters); native_compute_uploaded = front->compute_uploaded; @@ -1463,6 +1714,8 @@ void KeplerCompute::ReleaseFront() { native_macro = front->macro; front.reset(); front_macros = false; + // The engines schedule their QMDs on their own again. + GetFrontQmd().Lose(); LOG_INFO(HW_GPU, "GPU thread split stopped: compute macros back on the engine"); } @@ -1471,6 +1724,8 @@ void KeplerCompute::FrontApply(FrontState& fs, u32 method, u32 argument) { if (method == 0x45 || method == 0x47) { fs.compute_uploaded = true; } + Nx2ListIndirect::NoteUploadMethod(fs.upload, method, argument, + [&fs](u32 reg) { return fs.registers[reg]; }); fs.registers[method] = argument; switch (method) { case 0x45: @@ -1516,7 +1771,8 @@ void KeplerCompute::FrontCallMethod(Core::System& system, VideoCommon::GpuSplit& const GPUVAddr* segment) { auto& fs = Front(split); if (method >= fs.registers.size()) { - split.Method(this, method, argument, is_last_call, segment); + split.Method(this, method, argument, is_last_call, segment, FrontWrites(method)); + split.TagUnplaced(VideoCommon::UnplacedKind::Engine); // TEST return; } if (method >= 0xe00) { @@ -1534,14 +1790,87 @@ void KeplerCompute::FrontCallMethod(Core::System& system, VideoCommon::GpuSplit& return; } FrontApply(fs, method, argument); - split.Method(this, method, argument, is_last_call, segment); - if (method == 0x6c3) { - FrontNoteReport(split, fs, argument); + if (IsQmdMethod(method)) { + GetFrontQmd().MaybeTakeOver(split, native_qmd, QmdTakeOver::QmdMethod); + } + { + const VideoCommon::GpuSplit::PublishHold hold{split, ValuedFromMemory(method)}; + split.Method(this, method, argument, is_last_call, segment, FrontWrites(method)); + FrontNoteWrites(split, fs, method, argument, false); + } + FrontQmdCheck(split); +} + +// TEST (EDEN_NX2_DRAIN_CHECK): has the back compare what it reads and releases for the QMD +// methods just sent with what the front's QMD view read and noted for them, and now and then its +// QMD table with the view's. +void KeplerCompute::FrontQmdCheck(VideoCommon::GpuSplit& split) { + if (!VideoCommon::Nx2DrainCheck::g_enabled) [[likely]] { + return; + } + auto checks = GetFrontQmd().TakeChecks(); + if (!checks) { + return; + } + split.DeferFunction( + [predicted = std::move(checks->releases), reads = std::move(checks->reads), + table = std::move(checks->table), valued = checks->valued && QmdReleasesLandAtOnce(), + after_late = checks->table_after_late, wait_position = checks->wait_position, + first_waiting = checks->first_waiting, late = std::move(checks->late)] { + VideoCommon::Nx2DrainCheck::CheckQmdReleases(predicted, reads, valued, wait_position, + first_waiting); + VideoCommon::Nx2DrainCheck::CheckLateQmdReads(late); + if (table && valued) { + const NativeQmdScheduler shared; // The table the engines share + VideoCommon::Nx2DrainCheck::NoteQmdTables(table->SameStates(shared), + table->StateCount(), after_late); + } + }, + VideoCommon::GpuSplit::Writes::Noted); +} + +// What the back writes to guest memory running `method` with `argument`, just sent as +// FrontWrites tells: where reports and inline QMDs write, and what QMD launches release. +void KeplerCompute::FrontNoteWrites(VideoCommon::GpuSplit& split, const FrontState& fs, u32 method, + u32 argument, bool from_macro) { + switch (method) { + case 0x6c3: { + // ProcessNativeReportSemaphore: a payload with its timestamp at most. + const GPUVAddr address = + (GPUVAddr{fs.registers[0x6c0] & 0x1ffffff} << 32) | fs.registers[0x6c1]; + split.NoteWrite(memory_manager, address, 16); + FrontNoteReport(split, fs, argument, from_macro); + break; + } + case 0x107: + FrontInlineQmd(split, fs, argument); + break; + case KEPLER_COMPUTE_REG_INDEX(data_upload): { + // Upload::State::ProcessData, of the upload exec_upload started: the whole of it at each + // data method, at most. + Upload::Registers upload; + std::memcpy(&upload, fs.registers.data() + KEPLER_COMPUTE_REG_INDEX(upload), + sizeof(upload)); + const bool linear = (fs.registers[KEPLER_COMPUTE_REG_INDEX(exec_upload)] & 1) != 0; + const auto range = Upload::UploadedRange(upload, linear); + split.NoteWrite(memory_manager, range.address, range.size); + break; + } + case 0xae: + case 0xaf: + case 0xb0: + if (SignalsQmd(method, argument)) { + GetFrontQmd().Signal(split, FrontQmdMemory{memory_manager}, + GPUVAddr{fs.registers[0xad]} << 8, argument, nullptr); + } + break; + default: + break; } } -void KeplerCompute::FrontNoteReport(VideoCommon::GpuSplit& split, const FrontState& fs, - u32 control) { +void KeplerCompute::FrontNoteReport(VideoCommon::GpuSplit& split, const FrontState& fs, u32 control, + bool from_macro) { // SET_REPORT_SEMAPHORE_D was sent: what ProcessNativeReportSemaphore will write on the back. const u32 operation = control & 3; if (operation != 0 && operation != 2) { @@ -1553,12 +1882,69 @@ void KeplerCompute::FrontNoteReport(VideoCommon::GpuSplit& split, const FrontSta if (!device) { return; } - // A reduction combines the payload with the word's value then. + VideoCommon::Nx2AcquireStats::NoteRelease( + *device, VideoCommon::Nx2AcquireStats::Releaser::ReportCompute); + // A reduction combines the payload with the word's value then. The front knows that value + // when no compute or copy work sent since it may write the word (QMD releases): from the last + // release sent to the word if it is known, or from memory if nothing queued may change it. + // A macro's own launches count as work only after it ran (DmaPusher::NoteWriter). const bool reduction = ((control >> 3) & 1) != 0; - split.NoteSemaphoreWrite(*device, fs.registers[0x6c2], !reduction); - if (!reduction && ((control >> 28) & 1) == 0) { + if (reduction) { + const u32 payload = fs.registers[0x6c2]; + const u32 op = (control >> 9) & 7; + std::optional base; + if (!from_macro) { + // A fenced release lands once its fence ran, before or after the reduction. + if (const auto queued = split.QueuedSemaphoreWriteAt(*device)) { + if (queued->known && !queued->at_least && !queued->fenced && + queued->position >= split.LastWriterPosition()) { + base = queued->payload; + } + } else if (!split.WritersPending() && !Nx2MayBeGpuWritten(*device, sizeof(u32))) { + base = memory_manager.Read(address); + } + } + split.NoteSemaphoreWrite(*device, base ? ReduceSemaphore(op, false, *base, payload) : 0, + base.has_value()); + return; + } + // One word: a fence-flagged payload report (GpuSplit::FencedPayloadLanding). Else a payload + // report with a timestamp, written as the back runs it. + const bool one_word = ((control >> 28) & 1) != 0; + const auto landing = one_word ? VideoCommon::GpuSplit::FencedPayloadLanding() + : VideoCommon::GpuSplit::Landing::AtRun; + split.NoteSemaphoreWrite(*device, fs.registers[0x6c2], true, false, landing); + if (!one_word) { // 64 bits (a timestamp follows). - split.NoteSemaphoreWrite(*device + 4, 0, true); + split.NoteSemaphoreWrite(*device + 4, 0, true, false, landing); + } +} + +// LOAD_INLINE_QMD_DATA's last word (ProcessNativeMethod): the back writes the inline QMD to its +// address and launches it, unless the QMD waits for a PCAS to schedule it. +void KeplerCompute::FrontInlineQmd(VideoCommon::GpuSplit& split, const FrontState& fs, + u32 last_word) { + const GPUVAddr address = ((GPUVAddr{fs.registers[0xc6]} << 32) | fs.registers[0xc7]) << 8; + if (!memory_manager.IsFullyMappedRange(address, 256)) { + // The back writes nothing either, unless a change of the mappings still queued changes + // that. + if (!split.Ran(split.LastMappingChange())) { + GetFrontQmd().Lose(); + split.NoteUnplacedWrite(); + split.TagUnplaced(VideoCommon::UnplacedKind::InlineQmd); // TEST + } + return; + } + split.NoteWrite(memory_manager, address, 256); + // The words the back writes (registers 0xc8 to 0x107). + NativeQmdScheduler::Descriptor qmd; + std::memcpy(qmd.data(), fs.registers.data() + 0xc8, sizeof(qmd)); + qmd[63] = last_word; + if (const auto device = memory_manager.GpuToCpuAddress(address)) { + GetFrontQmd().NoteInlineWrite(split, *device, qmd); + } + if ((fs.registers[0xa7] & (1U << 24)) != 0 || (fs.registers[0xcc] & (1U << 12)) == 0) { + GetFrontQmd().Signal(split, FrontQmdMemory{memory_manager}, address, 3, &qmd); } } @@ -1572,17 +1958,66 @@ void KeplerCompute::FrontCallMultiMethod(Core::System& system, VideoCommon::GpuS } return; } - if (method < fs.registers.size()) { - for (u32 i = 0; i < amount; ++i) { - FrontApply(fs, method, base_start[i]); - } + if (method >= fs.registers.size()) { + split.MultiMethod(this, method, base_start, amount, methods_pending, segment, + FrontWrites(method)); + split.TagUnplaced(VideoCommon::UnplacedKind::Engine); // TEST + return; } - split.MultiMethod(this, method, base_start, amount, methods_pending, segment); - if (method == 0x6c3) { - for (u32 i = 0; i < amount; ++i) { - FrontNoteReport(split, fs, base_start[i]); + for (u32 i = 0; i < amount; ++i) { + FrontApply(fs, method, base_start[i]); + } + if (IsQmdMethod(method)) { + GetFrontQmd().MaybeTakeOver(split, native_qmd, QmdTakeOver::QmdMethod); + } + { + const VideoCommon::GpuSplit::PublishHold hold{split, ValuedFromMemory(method)}; + split.MultiMethod(this, method, base_start, amount, methods_pending, segment, + FrontWrites(method)); + // The back runs them one by one; what they write is the back's once it ran them all. + // Each upload data word may write the upload's range: once covers them. + const u32 notes = + method == KEPLER_COMPUTE_REG_INDEX(data_upload) ? (std::min)(amount, 1U) : amount; + for (u32 i = 0; i < notes; ++i) { + FrontNoteWrites(split, fs, method, base_start[i], false); } } + FrontQmdCheck(split); +} + +bool KeplerCompute::FrontTakeListWords(VideoCommon::GpuSplit& split, u32 method, u32 count, + u32 slot) { + if (!native_mode || native_data_ram == nullptr) { + return false; + } + auto& fs = Front(split); + if (fs.executing_macro || !fs.parameters.empty()) { + return false; // Not a call's first parameter + } + if (!list_front) { + list_front = std::make_unique(); + } + const auto positions = FrontMacroPositions(fs); + const u32 start = positions[((method - 0xe00) / 2) % positions.size()]; + const auto& macro = FrontMacroEngine(fs); + const auto no_emit = [](u32, u32) { return false; }; + std::vector no_params; + ListCallContext context{macro.native_code, + macro.native_code_generation, + start, + no_params, + std::span{*native_data_ram}, + fs.registers, + fs.upload, + split, + memory_manager, + rasterizer, + this, + fs.registers, + native_registers, + no_emit}; + return list_front->PrepareCall(context, Nx2ListIndirect::GetChannel(native_data_ram), slot, + count); } void KeplerCompute::FrontRunMacro(Core::System& system, VideoCommon::GpuSplit& split, @@ -1603,12 +2038,45 @@ void KeplerCompute::FrontRunMacro(Core::System& system, VideoCommon::GpuSplit& s return false; } FrontApply(fs, address, value); - split.MacroWrite(this, address, value); - if (address == 0x6c3) { - FrontNoteReport(split, fs, value); + if (IsQmdMethod(address)) { + GetFrontQmd().MaybeTakeOver(split, native_qmd, QmdTakeOver::QmdMethod); } + { + const VideoCommon::GpuSplit::PublishHold hold{split, ValuedFromMemory(address)}; + split.MacroWrite(this, address, value, FrontWrites(address)); + FrontNoteWrites(split, fs, address, value, true); + } + FrontQmdCheck(split); return true; }; + if (list_front && native_data_ram != nullptr) { + // GPU-written list words among the call's parameters or within its reach: a tagged run + // that leaves them to the back, or the words themselves first (Nx2ListIndirect). + ListCallContext context{macro.native_code, + macro.native_code_generation, + start, + fs.parameters, + std::span{*native_data_ram}, + fs.registers, + fs.upload, + split, + memory_manager, + rasterizer, + this, + fs.registers, + native_registers, + emit}; + bool concretized{}; + Nx2ListIndirect::Reason reason{}; + if (list_front->Run(context, Nx2ListIndirect::GetChannel(native_data_ram), concretized, + reason)) { + fs.parameters.clear(); + return; + } + if (concretized) { + Nx2ListIndirect::Stats::NoteConcretized(reason); // TEST + } + } #ifdef ARCHITECTURE_x86_64 if (!macro.native_jit) { macro.native_jit = std::make_shared(); diff --git a/src/video_core/engines/kepler_compute.h b/src/video_core/engines/kepler_compute.h index 6e99ffc239..f7240af5db 100644 --- a/src/video_core/engines/kepler_compute.h +++ b/src/video_core/engines/kepler_compute.h @@ -37,6 +37,10 @@ namespace Tegra { class MemoryManager; } +namespace Tegra::Nx2ListIndirect { +class ComputeFront; +} + namespace VideoCore { class RasterizerInterface; } @@ -251,6 +255,12 @@ class KeplerCompute final : public EngineInterface { return indirect_compute; } + /// The slot of GPU-written list words (Nx2ListIndirect) the dispatch being made takes its + /// group counts from, indirectly: its QMD's grid is the slot's placeholders. + std::optional GetIndirectListSlot() const { + return indirect_list_slot; + } + bool IsNativeMode() const { return native_mode; } bool ProcessGameDlss(u64 program_id, u64 native_key); /// Ocarina of Time with anti-aliasing at No sharpening (oot_network.h): true when this @@ -268,13 +278,22 @@ class KeplerCompute final : public EngineInterface { void FrontCallMultiMethod(Core::System& system, VideoCommon::GpuSplit& split, u32 method, const u32* base_start, u32 amount, u32 methods_pending, const GPUVAddr* segment); + /// The front, at a GPU-written list of `count` words that are all the parameters of the call + /// of macro method `method` about to start: whether the call takes the words as the + /// placeholders of `slot` (Nx2ListIndirect), the back's copy of them, instead of the words. + bool FrontTakeListWords(VideoCommon::GpuSplit& split, u32 method, u32 count, u32 slot); private: struct FrontState; FrontState& Front(VideoCommon::GpuSplit& split); void ReleaseFront(); void FrontApply(FrontState& state, u32 method, u32 argument); - void FrontNoteReport(VideoCommon::GpuSplit& split, const FrontState& state, u32 control); + void FrontNoteWrites(VideoCommon::GpuSplit& split, const FrontState& state, u32 method, + u32 argument, bool from_macro); + void FrontNoteReport(VideoCommon::GpuSplit& split, const FrontState& state, u32 control, + bool from_macro); + void FrontInlineQmd(VideoCommon::GpuSplit& split, const FrontState& state, u32 last_word); + void FrontQmdCheck(VideoCommon::GpuSplit& split); ///< TEST MacroEngine& FrontMacroEngine(FrontState& state); std::span FrontMacroPositions(FrontState& state); void FrontRunMacro(Core::System& system, VideoCommon::GpuSplit& split, FrontState& state); @@ -293,7 +312,9 @@ class KeplerCompute final : public EngineInterface { void CheckDlssNetwork(); [[nodiscard]] VideoCore::GameDlssReason GameDlssBlocker() const; void UpdateGameDlssStatus(bool complete, VideoCore::GameDlssReason blocker); - std::pair, std::vector> NativeImageHandles(GPUVAddr program); + /// The constant buffer 0 offsets of a program's image handles, read and stored: memoized + /// per program address, returned by reference (OoT asks for each of ~5,000 dispatches/s). + const std::pair, std::vector>& NativeImageHandles(GPUVAddr program); void RememberNativeUpscaleInput(); /// This frame's DLSS evaluation (Fortune's Weave): whether its passes all ran, and whether @@ -386,6 +407,9 @@ class KeplerCompute final : public EngineInterface { }; std::vector uploads; std::optional indirect_compute{}; + std::optional indirect_list_slot{}; + /// The front's GPU-written list words in this engine's macros (Nx2ListIndirect). + std::unique_ptr list_front; }; #define ASSERT_REG_POSITION(field_name, position) \ diff --git a/src/video_core/engines/maxwell_3d.cpp b/src/video_core/engines/maxwell_3d.cpp index e20d9dd266..241ba939c3 100644 --- a/src/video_core/engines/maxwell_3d.cpp +++ b/src/video_core/engines/maxwell_3d.cpp @@ -28,9 +28,11 @@ #include "video_core/gpu_split.h" #include "video_core/memory_manager.h" #include "video_core/nx2.h" -#include "video_core/nx2_async_readback.h" #include "video_core/nx2_cmdpage_diag.h" +#include "video_core/nx2_drain_check.h" #include "video_core/nx2_fast_draw.h" +#include "video_core/nx2_list_indirect.h" +#include "video_core/nx2_list_indirect_stats.h" #include "video_core/nx2_perf.h" #include "video_core/nx2_profile.h" #include "video_core/query_cache/frame_trace.h" @@ -62,8 +64,8 @@ Maxwell3D::Maxwell3D(MemoryManager& memory_manager_) execution_mask.reset(); for (size_t i = 0; i < execution_mask.size(); i++) execution_mask[i] = IsMethodExecutable(u32(i)); - fast_maxwell3d = !record_write_history && - !trace_first_method_writes && !VideoCommon::Nx2Profile::Enabled(); + // Profiled runs take the same path: CallMethod counts and times its calls itself. + fast_maxwell3d = !record_write_history && !trace_first_method_writes; } Maxwell3D::~Maxwell3D() = default; @@ -318,17 +320,8 @@ size_t Maxwell3D::EstimateIndexBufferSize() { return std::min(memory_manager.GetMemoryLayoutSize(start_address, byte_size * max_size) / byte_size, lower_cap); } -u32 Maxwell3D::ProcessShadowRam(u32 method, u32 argument) { - // Keep track of the register value in shadow_state when requested. - auto const c = shadow_state.shadow_ram_control; - if (c == Regs::ShadowRamControl::Track || c == Regs::ShadowRamControl::TrackWithFilter) - return shadow_state.reg_array[method] = argument; - else if (c == Regs::ShadowRamControl::Replay) - return shadow_state.reg_array[method]; - return argument; -} - void Maxwell3D::ConsumeSinkImpl(Core::System& system) { + const VideoCommon::Nx2Profile::FineScope profile_scope{VideoCommon::Nx2Profile::Zone::Sink}; if (front_macros && !VideoCommon::GpuSplit::OnBackThread()) { ReleaseFront(); } @@ -356,22 +349,6 @@ void Maxwell3D::ConsumeSinkImpl(Core::System& system) { method_sink.clear(); } -void Maxwell3D::ProcessDirtyRegisters(u32 method, u32 argument) { - // The instances after the first are drawn together once their loop ends, and they must see - // the state they were begun with: NVN's draw macros reset the base instance (and its copy in - // c[0][0x644]) right after the loop, before the macro returns. - if (draw_manager.HasDeferredDraw() && !DrawManager::IsInstanceLoopMethod(method)) [[unlikely]] { - draw_manager.DrawDeferred(*this, true); - } - if (VideoCommon::Nx2FastDraw::Enabled() && regs.reg_array[method] != argument && - !IsFastDrawParameter(method)) { - VideoCommon::Nx2FastDraw::Break(VideoCommon::Nx2FastDraw::Reason::Register, method); - } - regs.reg_array[method] = argument; - for (auto const& table : dirty.tables) - dirty.flags[table[method]] = true; -} - bool Maxwell3D::IsFastDrawParameter(u32 method) { switch (method) { case MAXWELL3D_REG_INDEX(const_buffer.size): @@ -380,6 +357,8 @@ bool Maxwell3D::IsFastDrawParameter(u32 method) { case MAXWELL3D_REG_INDEX(const_buffer.offset): case MAXWELL3D_REG_INDEX(global_base_vertex_index): case MAXWELL3D_REG_INDEX(global_base_instance_index): + // Written with the first vertex before each draw; nothing a draw resolves reads it. + case MAXWELL3D_REG_INDEX(vertex_id_base): case MAXWELL3D_REG_INDEX(index_buffer.start_addr_high): case MAXWELL3D_REG_INDEX(index_buffer.start_addr_low): case MAXWELL3D_REG_INDEX(index_buffer.limit_addr_high): @@ -388,7 +367,43 @@ bool Maxwell3D::IsFastDrawParameter(u32 method) { return true; default: return DrawManager::IsInstanceLoopMethod(method) || IsFastDrawDeltaMethod(method) || - IsFastDrawVertexBuffer(method); + IsFastDrawVertexBuffer(method) || IsFastDrawNeutralMethod(method) || + IsFastDrawReportOperand(method); + } +} + +bool Maxwell3D::IsFastDrawNeutralMethod(u32 method) { + switch (method) { + // A counter reset only touches the query cache. The render enable condition decides whether + // the next draws run (ShouldExecute, or host conditional rendering), not what they bind; a + // render pass either one ends is caught by the reusable draw's render pass check. + case MAXWELL3D_REG_INDEX(clear_report_value): + case MAXWELL3D_REG_INDEX(render_enable.mode): + // A report or semaphore release writes guest memory, which ProcessQueryGet notes as constant + // data: a delta draw that reads texture handles there takes the full path. + case MAXWELL3D_REG_INDEX(report_semaphore.query): + return true; + default: + return false; + } +} + +bool Maxwell3D::IsFastDrawReportOperand(u32 method) { + // SET_MME_MEM_ADDRESS_A/B and the data RAM address: only the macro engine's memory reads use + // them, and what a macro does with the words reaches the engine as writes of its own. + constexpr u32 MmeMemAddressA = 0x154; + constexpr u32 MmeDataRamAddress = 0x156; + if (method >= MmeMemAddressA && method <= MmeDataRamAddress) { + return true; + } + switch (method) { + // Only the report_semaphore.query method reads them. + case MAXWELL3D_REG_INDEX(report_semaphore.address_high): + case MAXWELL3D_REG_INDEX(report_semaphore.address_low): + case MAXWELL3D_REG_INDEX(report_semaphore.payload): + return true; + default: + return false; } } @@ -424,7 +439,8 @@ bool Maxwell3D::IsFastDrawDeltaMethod(u32 method) { } void Maxwell3D::ProcessMethodCall(u32 method, u32 argument, u32 nonshadow_argument, bool is_last_call) { - if (!DrawManager::IsInstanceLoopMethod(method) && !IsFastDrawDeltaMethod(method)) { + if (!DrawManager::IsInstanceLoopMethod(method) && !IsFastDrawDeltaMethod(method) && + !IsFastDrawNeutralMethod(method)) { VideoCommon::Nx2FastDraw::Break(VideoCommon::Nx2FastDraw::Reason::Method, method); } if (GetNativeMacroEngine().native_turing && method == MAXWELL3D_REG_INDEX(clear_surface)) { @@ -536,10 +552,8 @@ void Maxwell3D::TraceFirstMethodWrite(u32 method, u32 argument) { } } -void Maxwell3D::CallMethod(Core::System& system, u32 method, u32 method_argument, bool is_last_call) { - if (front_macros && !VideoCommon::GpuSplit::OnBackThread()) { - ReleaseFront(); - } +void Maxwell3D::DispatchMethod(Core::System& system, u32 method, u32 method_argument, + bool is_last_call) { if (!fast_maxwell3d || executing_macro != 0 || method >= MacroRegistersStart || method == MAXWELL3D_REG_INDEX(shadow_ram_control)) { return CallMethodSlow(system, method, method_argument, is_last_call); @@ -550,6 +564,39 @@ void Maxwell3D::CallMethod(Core::System& system, u32 method, u32 method_argument CallMethodFast(method, method_argument, is_last_call); } +void Maxwell3D::CallMethod(Core::System& system, u32 method, u32 method_argument, + bool is_last_call) { + if (front_macros && !VideoCommon::GpuSplit::OnBackThread()) { + ReleaseFront(); + } + // About half of the split's back's 3D writes (~5 million a second in Ocarina of Time) are state + // registers: CallMethodFast's state write, here without the calls to it. Not in + // execution_mask, they are not SET_SHADOW_RAM_CONTROL either. + if (fast_maxwell3d && executing_macro == 0 && method < MacroRegistersStart && + !execution_mask[method] && !perf_verify && + !VideoCommon::Nx2Profile::Enabled()) [[likely]] { + return WriteStateRegister(method, method_argument); + } + if (VideoCommon::Nx2Profile::Enabled()) [[unlikely]] { + return ProfiledCallMethod(system, method, method_argument, is_last_call); + } + DispatchMethod(system, method, method_argument, is_last_call); +} + +// EDEN_NX2_GPU_PROFILE: the path unprofiled runs take, counted. GpuSplit's back calls the 3D engine +// outside every zone (fast_maxwell3d), millions of times a second: a sample of those calls is +// timed (SampledScope). Called inside another zone (a back macro, a run of words, the unsplit GPU +// thread's method zone), the call is that zone's time, as before. +void Maxwell3D::ProfiledCallMethod(Core::System& system, u32 method, u32 method_argument, + bool is_last_call) { + const VideoCommon::Nx2Profile::SampledScope scope{VideoCommon::Nx2Profile::Zone::Method}; + if (method < MacroRegistersStart) { + VideoCommon::Nx2Profile::CountRegisterWrite(method, + regs.reg_array[method] == method_argument); + } + DispatchMethod(system, method, method_argument, is_last_call); +} + namespace { // Executable methods the draw manager does nothing for (ProcessDirtyRegisters is all they do). @@ -571,13 +618,10 @@ bool IsStoreOnlyExecutable(u32 method) { // no write history, trace or profile, not a macro call or SET_SHADOW_RAM_CONTROL. The hot // executable methods go straight to their handler. void Maxwell3D::CallMethodFast(u32 method, u32 method_argument, bool is_last_call) { - const u32 argument = ProcessShadowRam(method, method_argument); if (!execution_mask[method]) { - if (!SkipSameValueWrites() || regs.reg_array[method] != argument) { - ProcessDirtyRegisters(method, argument); - } - return; + return WriteStateRegister(method, method_argument); } + const u32 argument = ProcessShadowRam(method, method_argument); ProcessDirtyRegisters(method, argument); constexpr u32 cb_data = MAXWELL3D_REG_INDEX(const_buffer.buffer); if (method - cb_data < 16) { @@ -692,10 +736,18 @@ void Maxwell3D::CallConstBufferWords(Core::System& system, const u32* methods, } return; } + // Counted and timed as CallMethod counts and times each word (EDEN_NX2_GPU_PROFILE). + const VideoCommon::Nx2Profile::SampledScope scope{VideoCommon::Nx2Profile::Zone::Method, count}; + const bool profile = VideoCommon::Nx2Profile::Enabled(); // As CallMethodFast for each word: the first may issue a deferred draw, before any upload. for (u32 i = 0; i < count; ++i) { + if (profile) [[unlikely]] { + VideoCommon::Nx2Profile::CountRegisterWrite(methods[i], + regs.reg_array[methods[i]] == arguments[i]); + } ProcessDirtyRegisters(methods[i], ProcessShadowRam(methods[i], arguments[i])); } + const VideoCommon::Nx2Profile::FineScope upload_scope{VideoCommon::Nx2Profile::Zone::CbUpload}; if (VideoCommon::Nx2FastDraw::Enabled()) { NoteFastDrawConstantData(address + offset, address + offset + bytes); } @@ -726,14 +778,6 @@ void Maxwell3D::CallMethodSlow(Core::System& system, u32 method, u32 method_argu } ASSERT(method < Regs::NUM_REGS && "Invalid Maxwell3D register, increase the size of the Regs structure"); - if (VideoCommon::Nx2Profile::Enabled()) [[unlikely]] { - VideoCommon::Nx2Profile::Add(VideoCommon::Nx2Profile::Counter::MethodWrites, 1); - const bool same = regs.reg_array[method] == method_argument; - if (same) { - VideoCommon::Nx2Profile::Add(VideoCommon::Nx2Profile::Counter::SameValueWrites, 1); - } - VideoCommon::Nx2Profile::CountMethod(method, same); - } if (method == MAXWELL3D_REG_INDEX(shadow_ram_control) && front_macros) [[unlikely]] { back_control_writes[back_control_count++ % back_control_writes.size()] = { method_argument, static_cast(shadow_state.shadow_ram_control), is_last_call ? 1U : 0U}; @@ -868,6 +912,8 @@ void Maxwell3D::StampQueryResult(Core::System& system, u64 payload, bool long_qu } void Maxwell3D::ProcessQueryGet() { + const VideoCommon::Nx2Profile::FineScope profile_scope{ + VideoCommon::Nx2Profile::Zone::QueryReport}; VideoCommon::QueryPropertiesFlags flags{}; if (regs.report_semaphore.query.short_query == 0) { flags |= VideoCommon::QueryPropertiesFlags::HasTimeout; @@ -877,6 +923,12 @@ void Maxwell3D::ProcessQueryGet() { static_cast(regs.report_semaphore.query.report.Value()); const u32 payload = regs.report_semaphore.payload; const u32 subreport = regs.report_semaphore.query.sub_report; + if (VideoCommon::Nx2FastDraw::Enabled()) { + // The release lands in guest memory: a short one writes the payload, a long one the + // 16-byte report. + const u64 bytes = regs.report_semaphore.query.short_query != 0 ? 4 : 16; + NoteFastDrawConstantData(sequence_address, sequence_address + bytes); + } switch (regs.report_semaphore.query.operation) { case Regs::ReportSemaphore::Operation::Release: if (regs.report_semaphore.query.short_query != 0) { @@ -902,6 +954,8 @@ void Maxwell3D::ProcessQueryGet() { } void Maxwell3D::ProcessQueryCondition() { + const VideoCommon::Nx2Profile::FineScope profile_scope{ + VideoCommon::Nx2Profile::Zone::RenderEnable}; if (rasterizer->AccelerateConditionalRendering()) { execute_on = true; return; @@ -968,6 +1022,8 @@ void Maxwell3D::ProcessQueryCondition() { } void Maxwell3D::ProcessCounterReset() { + const VideoCommon::Nx2Profile::FineScope profile_scope{ + VideoCommon::Nx2Profile::Zone::CounterReset}; const auto query_type = [clear_report = regs.clear_report_value]() { switch (clear_report) { case Tegra::Engines::Maxwell3D::Regs::ClearReport::ZPassPixelCount: @@ -996,6 +1052,7 @@ void Maxwell3D::ProcessSyncPoint() { } void Maxwell3D::ProcessCBBind(size_t stage_index) { + const VideoCommon::Nx2Profile::FineScope profile_scope{VideoCommon::Nx2Profile::Zone::CbBind}; // Bind the buffer currently in CB_ADDRESS to the specified index in the desired shader // stage. const auto& bind_data = regs.bind_groups[stage_index]; @@ -1066,6 +1123,7 @@ void Maxwell3D::NoteFastDrawConstantData(GPUVAddr begin, GPUVAddr end) { } void Maxwell3D::ProcessCBMultiData(const u32* start_base, u32 amount) { + const VideoCommon::Nx2Profile::FineScope profile_scope{VideoCommon::Nx2Profile::Zone::CbUpload}; // Write the input value to the current const buffer at the current position. const GPUVAddr buffer_address = regs.const_buffer.Address(); ASSERT(buffer_address != 0); @@ -1175,7 +1233,9 @@ struct Maxwell3D::FrontState { std::vector writes; ///< The registers its code can write std::bitset write_set; ///< The same, for the back's check std::atomic escaped{}; ///< The back saw it write another register + u32 checked{}; ///< Calls EDEN_NX2_BACK_MACRO_VERIFY checked }; + std::vector> back_record; ///< A checked call's writes on the back /// By start in the macro code of back_generation, once called. std::vector> back_macros; u64 back_generation{~u64{0}}; @@ -1234,9 +1294,9 @@ void CheckDmaRead(u32 macro, GPUVAddr source, bool drained, bool waited, } last_report = now; LOG_INFO(HW_GPU, - "Macro DMA read checks (EDEN_NX2_DMA_READ_VERIFY) in {:.1f} s: {} reads, {} " - "without draining the split ({} waited for queued writers), {} of those with other " - "words than after a drain ({} in all)", + "Macro DMA read checks in {:.1f} s: {} reads, {} without draining the split ({} " + "waited for queued writers), {} of those with other words than after a drain ({} " + "in all)", seconds, reads, undrained_reads, waits, differences, total_differences); reads = 0; undrained_reads = 0; @@ -1288,6 +1348,15 @@ bool FrontSkipSame() { return skip; } +// EDEN_NX2_BACK_MACROS=0: macros that read memory run on the front too (FrontRunOnBack). +bool BackMacrosEnabled() { + static const bool enabled = [] { + const char* value = std::getenv("EDEN_NX2_BACK_MACROS"); + return value == nullptr || value[0] != '0'; + }(); + return enabled; +} + // Whether a macro the split's back runs may write `method` (FrontRunOnBack): the front then only // loses track of the register's value. Not the methods with effects the front tracks (macro code // and bindings, the shadow RAM mode, draws, constant buffer data, reports, uploads, syncpoints), @@ -1326,18 +1395,94 @@ bool BackWritable(const Maxwell3D& maxwell3d, u32 method) { } } -// split.WaitUntil for a macro the back runs in the front's place (FrontRunOnBack), counted for the -// async read-back line (EDEN_NX2_GPU_PROFILE). -bool WaitForBackMacros(VideoCommon::GpuSplit& split, u64 position, VideoCommon::DrainSite site) { - if (!VideoCommon::Nx2Profile::Enabled()) { - return split.WaitUntil(position, site); +// EDEN_NX2_BACK_MACRO_VERIFY= (64 without a number): the first n calls of each macro the back +// runs (FrontRunOnBack) also run the old way, on the front once the split is drained +// (FrontRunTheOldWay), and the front compares the writes (method, value) both make; they must +// not differ. Every 10 s it logs the counts, with the front's reads and replays of registers +// such macros wrote and how many of them waited for the back. +u32 BackMacroChecks() { + static const u32 checks = [] { + const char* value = std::getenv("EDEN_NX2_BACK_MACRO_VERIFY"); + if (value == nullptr) { + return 0U; + } + const unsigned long calls = std::strtoul(value, nullptr, 10); + return calls != 0 ? static_cast(calls) : 64U; + }(); + return checks; +} + +struct BackMacroStats { + u64 calls{}; + u64 checked{}; + u64 differences{}; + u64 total_differences{}; + u64 reads{}; + u64 read_waits{}; + u64 replays{}; + u64 replay_waits{}; + std::chrono::steady_clock::time_point last_report{}; +}; + +BackMacroStats& BackStats() { + static BackMacroStats stats; + return stats; +} + +void ReportBackMacroStats() { + auto& stats = BackStats(); + const auto now = std::chrono::steady_clock::now(); + if (stats.last_report == std::chrono::steady_clock::time_point{}) { + stats.last_report = now; + return; + } + const double seconds = std::chrono::duration(now - stats.last_report).count(); + if (seconds < 10.0) { + return; } - const u64 start = VideoCore::AsyncReadback::Now(); - const bool waited = split.WaitUntil(position, site); - if (waited) { - VideoCore::AsyncReadback::CountBackWait(VideoCore::AsyncReadback::Now() - start); + LOG_INFO(HW_GPU, + "Split back macros in {:.1f} s: {} calls, {} checked against the old way, {} of " + "those with other writes ({} in all); {} front reads of registers they wrote ({} " + "waited), {} replays of their shadow RAM entries ({} waited)", + seconds, stats.calls, stats.checked, stats.differences, stats.total_differences, + stats.reads, stats.read_waits, stats.replays, stats.replay_waits); + stats = BackMacroStats{.total_differences = stats.total_differences, .last_report = now}; +} + +std::string DescribeWrites(std::span> writes) { + std::string text; + for (size_t i = 0; i < std::min(writes.size(), 12); ++i) { + text += fmt::format("{}{:#x}={:#x}", i == 0 ? "" : ",", writes[i].first, writes[i].second); + } + return writes.size() > 12 ? text + ",..." : text; +} + +// GPU-written list words the channel's compute macros left in its data RAM (Nx2ListIndirect): +// the front runs 3D macros on the words themselves, so a macro at `start` that may read one +// gets them first, read back from their slots. +void ConcretizeListWords(VideoCommon::GpuSplit& split, VideoCore::RasterizerInterface* rasterizer, + Nx2ListIndirect::DataRamWords& channel, std::span data_ram, + const MacroEngine& engine, u32 start) { + const auto& reads = + channel.reads_3d.Get(engine.native_code, engine.native_code_generation, start); + if (!Nx2ListIndirect::MayRead(reads.data_ram_known, reads.data_ram, channel.words)) { + return; } - return waited; + const auto release = [&](u32 slot) { Nx2ListIndirect::ReleaseSlot(split, rasterizer, slot); }; + Nx2ListIndirect::Prune(channel.words, [data_ram](u32 word) { return data_ram[word]; }, release); + if (!Nx2ListIndirect::MayRead(reads.data_ram_known, reads.data_ram, channel.words)) { + return; + } + Nx2ListIndirect::Stats::NoteConcretized(Nx2ListIndirect::Reason::Macro3d); // TEST + Nx2ListIndirect::ConcretizeDataRam( + channel.words, data_ram, + [&](std::span slots, std::span words) { + const u64 read_start = Nx2ListIndirect::Stats::NowNs(); // TEST + Nx2ListIndirect::ReadSlots(split, rasterizer, slots, words); + Nx2ListIndirect::Stats::NoteRead(slots.size(), + Nx2ListIndirect::Stats::NowNs() - read_start); + }, + release); } } // Anonymous namespace @@ -1373,6 +1518,19 @@ Maxwell3D::FrontState& Maxwell3D::Front(Core::System& system, VideoCommon::GpuSp // split stopped (accuracy settings changed mid-game). Hand the macros back: the front's copy // goes stale from now on, and a restart copies the registers again (Front). void Maxwell3D::ReleaseFront() { + // The macros run on the engine from now on: the data RAM gets the GPU-written list words + // its words hold placeholders of (Nx2ListIndirect). + if (auto* channel = Nx2ListIndirect::FindChannel(&native_data_ram); + channel != nullptr && !channel->words.empty()) { + auto& split = VideoCommon::GpuSplit::Get(); + Nx2ListIndirect::Stats::NoteConcretized(Nx2ListIndirect::Reason::Stop); // TEST + Nx2ListIndirect::ConcretizeDataRam( + channel->words, std::span{native_data_ram}, + [&](std::span slots, std::span words) { + Nx2ListIndirect::ReadSlots(split, rasterizer, slots, words); + }, + [&](u32 slot) { Nx2ListIndirect::ReleaseSlot(split, rasterizer, slot); }); + } executing_macro = front->executing_macro; macro_params = std::move(front->parameters); // Entries macros the back ran may have written are the engine's already. @@ -1494,10 +1652,20 @@ void Maxwell3D::FrontNoteReport(VideoCommon::GpuSplit& split, const FrontState& return; } const bool payload = report.query.report == Regs::ReportSemaphore::Report::Payload; - split.NoteSemaphoreWrite(*device, report.payload, payload); + // As QueryCacheBase::CounterReport writes it: a payload report without a fence flag as the + // back runs it, a short (fence-flagged) payload release as FencedPayloadLanding says, a + // counter once a fence ran. + const bool fence_flagged = operation == Regs::ReportSemaphore::Operation::Release && + report.query.short_query != 0; + const auto landing = !payload ? VideoCommon::GpuSplit::Landing::Fenced + : fence_flagged ? VideoCommon::GpuSplit::FencedPayloadLanding() + : VideoCommon::GpuSplit::Landing::AtRun; + split.NoteSemaphoreWrite(*device, report.payload, payload, false, landing); + VideoCommon::Nx2AcquireStats::NoteRelease(*device, + VideoCommon::Nx2AcquireStats::Releaser::Report3D); if (report.query.short_query == 0) { // 64 bits (a timestamp follows). - split.NoteSemaphoreWrite(*device + 4, 0, payload); + split.NoteSemaphoreWrite(*device + 4, 0, payload, false, landing); } } @@ -1628,6 +1796,10 @@ void Maxwell3D::FrontRunMacro(Core::System& system, VideoCommon::GpuSplit& split VideoCommon::Nx2Profile::CountMethod(method, false); if (!FrontRunOnBack(system, split, fs, engine, start)) { VideoCommon::Nx2Profile::Scope profile_scope{VideoCommon::Nx2Profile::Zone::Macro}; + if (auto* channel = Nx2ListIndirect::FindChannel(&native_data_ram); + channel != nullptr && !channel->words.empty()) { + ConcretizeListWords(split, rasterizer, *channel, native_data_ram, engine, start); + } engine.native_interpreter.active_dram = &native_data_ram; const auto parameter_count = fs.parameters.size(); const auto read = [&](u32 address) { @@ -1667,8 +1839,9 @@ void Maxwell3D::FrontRunMacro(Core::System& system, VideoCommon::GpuSplit& split fs.parameters.clear(); if (fs.draw_pending) { fs.draw_pending = false; + // A deferred draw. split.Defer([this, system_ptr = &system] { FinishFrontMacro(*system_ptr); }, - VideoCommon::GpuSplit::FastDrawNeutral); + VideoCommon::GpuSplit::Writes::Caches, VideoCommon::GpuSplit::FastDrawNeutral); } if (FrontVerifyEnabled() && (++fs.calls & 4095) == 0) [[unlikely]] { FrontVerify(split, fs); @@ -1685,7 +1858,7 @@ bool Maxwell3D::FrontRunOnBack(Core::System& system, VideoCommon::GpuSplit& spli const MacroEngine& engine, u32 start) { using DrainSite = VideoCommon::DrainSite; using BackMacro = FrontState::BackMacro; - if (fs.parameters.size() > 0x1000) { + if (!BackMacrosEnabled() || fs.parameters.size() > 0x1000) { return false; } // A macro the back ran wrote another register: the front's copies may be stale anywhere. It @@ -1704,9 +1877,9 @@ bool Maxwell3D::FrontRunOnBack(Core::System& system, VideoCommon::GpuSplit& spli } FrontVerify(split, fs); }; - if (fs.back_generation != engine.native_code_generation) { + if (fs.back_generation != engine.native_code_generation) [[unlikely]] { // The macros sent before run the code they were sent with, and check their records. - WaitForBackMacros(split, fs.back_last, DrainSite::BackCode); + split.WaitForPosition(fs.back_last, DrainSite::BackCode); if (fs.back_escapes.load(std::memory_order_relaxed) != fs.back_escapes_seen) { take_escapes(); } @@ -1714,14 +1887,15 @@ bool Maxwell3D::FrontRunOnBack(Core::System& system, VideoCommon::GpuSplit& spli fs.back_macros.resize(engine.native_code.size() / 3); fs.back_generation = engine.native_code_generation; fs.back_code_copied = false; - } else if (fs.back_escapes.load(std::memory_order_relaxed) != fs.back_escapes_seen) { + } else if (fs.back_escapes.load(std::memory_order_relaxed) != fs.back_escapes_seen) + [[unlikely]] { take_escapes(); } if (start >= fs.back_macros.size()) { return false; } auto& known = fs.back_macros[start]; - if (!known) { + if (!known) [[unlikely]] { known = std::make_unique(); known->start = start; const auto effects = TuringMacroInterpreter::AnalyzeEffects(engine.native_code, start); @@ -1771,27 +1945,30 @@ bool Maxwell3D::FrontRunOnBack(Core::System& system, VideoCommon::GpuSplit& spli back.native_code_generation = engine.native_code_generation; fs.back_code_copied = true; } + const bool check = known->checked < BackMacroChecks(); + std::vector> old_writes; + if (check) [[unlikely]] { + ++known->checked; + old_writes = FrontRunTheOldWay(split, fs, start); + fs.back_record.clear(); + back.split_record = &fs.back_record; + } split.DeferWords( [this, system_ptr = &system, engine_ptr = &back, macro = known.get(), escapes = &fs.back_escapes](std::span parameters) { // As ProcessMacro calls a macro, with the back's engine. - namespace AsyncReadback = VideoCore::AsyncReadback; - const bool profiled = VideoCommon::Nx2Profile::Enabled(); - const u64 start = profiled ? AsyncReadback::Now() : 0; ConsumeSink(*system_ptr); engine_ptr->split_writes = ¯o->write_set; const u64 unexpected = engine_ptr->split_unexpected_writes; engine_ptr->Execute(*system_ptr, *this, macro->start, parameters); - if (engine_ptr->split_unexpected_writes != unexpected) { + if (engine_ptr->split_unexpected_writes != unexpected) [[unlikely]] { macro->escaped.store(true, std::memory_order_relaxed); escapes->fetch_add(1, std::memory_order_release); } - if (profiled) { - AsyncReadback::CountBackMacro(AsyncReadback::Now() - start); - AsyncReadback::MaybeReport(); - } }, - fs.parameters); + // Its writes (BackWritable) touch registers and the caches, not guest memory. + fs.parameters, VideoCommon::GpuSplit::Writes::Caches); + fs.sink_pending = false; // The back macro consumes the sink before it runs. const u64 position = split.Position(); fs.back_last = position; const bool tracked = fs.control == Regs::ShadowRamControl::Track || @@ -1803,16 +1980,115 @@ bool Maxwell3D::FrontRunOnBack(Core::System& system, VideoCommon::GpuSplit& spli } fs.back_owned[written] = true; } + if (BackMacroChecks() != 0) [[unlikely]] { + auto& stats = BackStats(); + ++stats.calls; + if (check) { + split.Drain(DrainSite::FrontVerify); + back.split_record = nullptr; + ++stats.checked; + if (old_writes != fs.back_record) { + ++stats.differences; + if (++stats.total_differences <= 16) { + LOG_WARNING(HW_GPU, + "Split back macro check: macro at {:#x} wrote {} the old way, {} " + "on the back", + start, DescribeWrites(old_writes), DescribeWrites(fs.back_record)); + } + } + } + ReportBackMacroStats(); + } return true; } +// EDEN_NX2_BACK_MACRO_VERIFY: the writes (method, value) the call FrontRunOnBack sends makes run +// the old way: on the front, against its copies of the registers, with the split drained first +// and its DMA reads made once the caches wrote the words out. Registers macros the back ran may +// have written are the back's, exact once drained; the call's own writes are what the back's +// CallMethod leaves (the shadow RAM applied). Writes nothing itself. +std::vector> Maxwell3D::FrontRunTheOldWay(VideoCommon::GpuSplit& split, + const FrontState& fs, u32 start) { + split.Drain(VideoCommon::DrainSite::FrontVerify); + std::vector> writes; + std::unordered_map values; + std::unordered_map entries; + const auto value_of = [&](u32 method) { + const auto it = values.find(method); + return it != values.end() ? it->second + : fs.back_regs[method] != 0 ? regs.reg_array[method] + : fs.regs[method]; + }; + const auto entry_of = [&](u32 method) { + const auto it = entries.find(method); + return it != entries.end() ? it->second + : fs.back_shadow[method] != 0 ? shadow_state.reg_array[method] + : fs.shadow[method]; + }; + std::vector fifo = fs.parameters; + const auto read = [&](u32 address) { + return address < Regs::NUM_REGS ? value_of(address) : 0U; + }; + const auto emit = [&](u32 address, u32 value) { + if (address >= MacroRegistersStart) { + return false; + } + writes.emplace_back(address, value); + u32 stored = value; + if (fs.control == Regs::ShadowRamControl::Track || + fs.control == Regs::ShadowRamControl::TrackWithFilter) { + entries[address] = value; + } else if (fs.control == Regs::ShadowRamControl::Replay) { + stored = entry_of(address); + } + values[address] = stored; + if (address == 0x158) { + // As MacroEngine::Execute reads on the back. + const GPUVAddr source = (GPUVAddr{value_of(0x154) & 0x1ffffff} << 32) | value_of(0x155); + const u32 words = std::min(value, 0x1000); + const size_t read_bytes = size_t{words} * sizeof(u32); + split.Run( + [this, source, read_bytes] { + VideoCommon::QueryDma::FlushPending(memory_manager, *rasterizer, source, + read_bytes); + if (memory_manager.IsMemoryDirty(source, read_bytes, + VideoCommon::CacheType::BufferCache)) { + memory_manager.FlushRegion(source, read_bytes, + VideoCommon::CacheType::BufferCache); + } + }, + VideoCommon::DrainSite::FrontVerify); + for (u32 i = 0; i < words; ++i) { + fifo.push_back(memory_manager.Read(source + u64{i} * sizeof(u32))); + } + } + return true; + }; + auto& engine = GetNativeMacroEngine(); +#ifdef ARCHITECTURE_x86_64 + if (!engine.native_jit) { + engine.native_jit = std::make_shared(); + } + engine.native_jit->Prepare(engine.native_code, engine.native_code_generation); + engine.native_jit->Execute(start, fifo, std::span{native_data_ram}, read, emit); +#else + engine.native_interpreter.Execute(engine.native_code, start, fifo, read, emit); +#endif + return writes; +} + // A register as the back holds it once it ran what the front sent: the front's copy, or the // back's once it ran the last macro there that can write it. No entry sent after that macro // writes the register (a front write takes it over first, a later macro there would be the one // waited for), so the back's value stays until the front writes the register again. u32 Maxwell3D::FrontRegister(VideoCommon::GpuSplit& split, FrontState& fs, u32 method) { if (fs.back_regs[method] != 0) { - WaitForBackMacros(split, fs.back_regs[method], VideoCommon::DrainSite::BackRegister); + const bool waited = + split.WaitForPosition(fs.back_regs[method], VideoCommon::DrainSite::BackRegister); + if (BackMacroChecks() != 0) [[unlikely]] { + ++BackStats().reads; + BackStats().read_waits += waited ? 1 : 0; + } fs.regs[method] = regs.reg_array[method]; fs.back_regs[method] = 0; fs.back_owned[method] = fs.back_shadow[method] != 0; @@ -1828,7 +2104,12 @@ bool Maxwell3D::FrontTakeOver(VideoCommon::GpuSplit& split, FrontState& fs, u32 using Control = Regs::ShadowRamControl; const bool tracked = fs.control == Control::Track || fs.control == Control::TrackWithFilter; if (fs.back_shadow[method] != 0 && fs.control == Control::Replay) { - WaitForBackMacros(split, fs.back_shadow[method], VideoCommon::DrainSite::BackRegister); + const bool waited = + split.WaitForPosition(fs.back_shadow[method], VideoCommon::DrainSite::BackRegister); + if (BackMacroChecks() != 0) [[unlikely]] { + ++BackStats().replays; + BackStats().replay_waits += waited ? 1 : 0; + } fs.shadow[method] = shadow_state.reg_array[method]; fs.back_shadow[method] = 0; } @@ -1895,9 +2176,9 @@ void Maxwell3D::FrontNoteWrites(VideoCommon::GpuSplit& split, const FrontState& }; if (fs.draw_pending) { // A draw, or one an instance loop deferred to this entry (ProcessDirtyRegisters): where - // its shaders store, and what the caches download for it, only the back knows. - split.NoteAnyWrite(); - return; + // its shaders store, and what the caches download for it, only the back knows. Guest + // memory it writes through the caches only. + split.NoteCacheWrite(); } if (method >= cb_data && method < cb_data + 16) { // FrontWrite and FrontCallMultiMethod advanced the offset past these words already. @@ -1914,18 +2195,15 @@ void Maxwell3D::FrontNoteWrites(VideoCommon::GpuSplit& split, const FrontState& 16); break; case MAXWELL3D_REG_INDEX(inline_data): { - // Upload::State::ProcessData: lines of line_length_in bytes, pitch apart. + // Upload::State::ProcessData: lines of line_length_in bytes, pitch apart, or swizzled + // into the destination surface. const auto launch = std::bit_cast(fs.regs[MAXWELL3D_REG_INDEX(launch_dma)]); - const u32 lines = fs.regs[MAXWELL3D_REG_INDEX(upload.line_count)]; - const u32 length = fs.regs[MAXWELL3D_REG_INDEX(upload.line_length_in)]; - if (launch.memory_layout != Regs::LaunchDMA::Layout::Pitch) { - split.NoteAnyWrite(); // Swizzled into the destination surface - } else if (lines != 0 && length != 0) { - split.NoteWrite(memory_manager, address(MAXWELL3D_REG_INDEX(upload.dest.address_high)), - u64{fs.regs[MAXWELL3D_REG_INDEX(upload.dest.pitch)]} * (lines - 1) + - length); - } + Upload::Registers upload; + std::memcpy(&upload, fs.regs.data() + MAXWELL3D_REG_INDEX(upload), sizeof(upload)); + const auto range = + Upload::UploadedRange(upload, launch.memory_layout == Regs::LaunchDMA::Layout::Pitch); + split.NoteWrite(memory_manager, range.address, range.size); break; } case MAXWELL3D_REG_INDEX(render_enable.mode): { @@ -1940,13 +2218,14 @@ void Maxwell3D::FrontNoteWrites(VideoCommon::GpuSplit& split, const FrontState& (mode == Regs::RenderEnable::Mode::Conditional || mode == Regs::RenderEnable::Mode::IfEqual || mode == Regs::RenderEnable::Mode::IfNotEqual)) { - split.NoteAnyWrite(); + split.NoteCacheWrite(); } break; } case MAXWELL3D_REG_INDEX(sync_info): - // A syncpoint increment: macros read syncpoints through memory too (MemoryManager::Read). - split.NoteAnyWrite(); + // A syncpoint increment: macros read syncpoints through memory too (MemoryManager::Read + // of the syncpoint shim), guest memory holds nothing new. + split.NoteCacheWrite(); break; default: break; @@ -1999,8 +2278,6 @@ void Maxwell3D::FrontDmaRead(VideoCommon::GpuSplit& split, FrontState& fs, u32 m }, VideoCommon::DrainSite::MacroDmaFlush); }; - namespace AsyncReadback = VideoCore::AsyncReadback; - const u64 wait_start = AsyncReadback::Now(); // Only the work still queued that can write the words has to run first (FrontNoteWrites, // GpuSplit::WaitForWrites). Past it no entry can make them GPU-written or add reports there, // so the caches have nothing to write out unless they already hold it now. @@ -2011,17 +2288,9 @@ void Maxwell3D::FrontDmaRead(VideoCommon::GpuSplit& split, FrontState& fs, u32 m if (drained) { write_out(); } - AsyncReadback::CountDmaRead(waited || drained, drained, AsyncReadback::Now() - wait_start); - if (VideoCommon::Nx2Profile::Enabled()) { - AsyncReadback::MaybeReport(); - } - // EDEN_NX2_DMA_READ_VERIFY: a read made without draining the split drains it anyway and has - // the caches write the words out, as a full drain would, and the macro gets those words. - // CheckDmaRead counts the reads whose words differ from the ones read first. - const bool verify = AsyncReadback::DmaReadVerify(); std::vector undrained; std::vector checked; - if (verify) { + if (dma_read_verify) [[unlikely]] { if (!drained) { for (u32 i = 0; i < words; ++i) { undrained.push_back(memory_manager.Read(source + u64{i} * sizeof(u32))); @@ -2041,11 +2310,11 @@ void Maxwell3D::FrontDmaRead(VideoCommon::GpuSplit& split, FrontState& fs, u32 m if (!diag_before.empty() && diag_before[i] != word) { ++changed; } - if (verify) { + if (dma_read_verify) [[unlikely]] { checked.push_back(word); } } - if (verify) { + if (dma_read_verify) [[unlikely]] { CheckDmaRead(macro, source, drained, waited, undrained, checked); } if (Tegra::Nx2CmdPageDiag::g_enabled) { @@ -2129,6 +2398,7 @@ void Maxwell3D::FrontVerify(VideoCommon::GpuSplit& split, FrontState& fs) { } void Maxwell3D::FinishFrontMacro(Core::System& system) { + const VideoCommon::Nx2Profile::FineScope profile_scope{VideoCommon::Nx2Profile::Zone::MacroEnd}; // What ProcessMacro and CallMacroMethod do around the macro. ConsumeSink(system); draw_manager.DrawDeferred(*this); diff --git a/src/video_core/engines/maxwell_3d.h b/src/video_core/engines/maxwell_3d.h index 5b9ba1b1a1..21081cd585 100644 --- a/src/video_core/engines/maxwell_3d.h +++ b/src/video_core/engines/maxwell_3d.h @@ -3280,6 +3280,11 @@ class Maxwell3D final : public EngineInterface { static bool IsFastDrawDeltaMethod(u32 method); /// A vertex stream's address, size or limit (not its stride or enable). static bool IsFastDrawVertexBuffer(u32 method); + /// Executable methods whose work leaves what a draw resolves to alone (counter resets, the + /// render enable condition): fast draw records nothing for them. + static bool IsFastDrawNeutralMethod(u32 method); + /// Registers only a report or a macro memory read takes as operands. + static bool IsFastDrawReportOperand(u32 method); void NoteFastDrawConstantData(GPUVAddr begin, GPUVAddr end); /// Handles a write to the CB_DATA[i] register. @@ -3293,14 +3298,58 @@ class Maxwell3D final : public EngineInterface { void ProcessMacro(Core::System& system, u32 method, const u32* base_start, u32 amount, bool is_last_call); - u32 ProcessShadowRam(u32 method, u32 argument); + /// The value a write leaves in the register, as the shadow RAM mode has it: the argument + /// (tracked in shadow_state when tracking), or the shadow copy when replaying. + u32 ProcessShadowRam(u32 method, u32 argument) { + const auto control = shadow_state.shadow_ram_control; + if (control == Regs::ShadowRamControl::Track || + control == Regs::ShadowRamControl::TrackWithFilter) { + return shadow_state.reg_array[method] = argument; + } + if (control == Regs::ShadowRamControl::Replay) { + return shadow_state.reg_array[method]; + } + return argument; + } + + /// Stores a register's value and sets its dirty flags. Defined here, as the hot method paths + /// call it for nearly every write. + void ProcessDirtyRegisters(u32 method, u32 argument) { + // The instances after the first are drawn together once their loop ends, and they must + // see the state they were begun with: NVN's draw macros reset the base instance (and its + // copy in c[0][0x644]) right after the loop, before the macro returns. + if (draw_manager.HasDeferredDraw() && !DrawManager::IsInstanceLoopMethod(method)) + [[unlikely]] { + draw_manager.DrawDeferred(*this, true); + } + if (VideoCommon::Nx2FastDraw::Enabled() && regs.reg_array[method] != argument && + !IsFastDrawParameter(method)) { + VideoCommon::Nx2FastDraw::Break(VideoCommon::Nx2FastDraw::Reason::Register, method); + } + regs.reg_array[method] = argument; + for (const auto& table : dirty.tables) { + dirty.flags[table[method]] = true; + } + } - void ProcessDirtyRegisters(u32 method, u32 argument); + /// CallMethodFast for a state register (not in execution_mask): a write of the value the + /// register holds changes nothing (EDEN_NX2_SKIP_SAME=0 keeps it). + void WriteStateRegister(u32 method, u32 method_argument) { + const u32 argument = ProcessShadowRam(method, method_argument); + if (!SkipSameValueWrites() || regs.reg_array[method] != argument) { + ProcessDirtyRegisters(method, argument); + } + } void ConsumeSinkImpl(Core::System& system) override; void ProcessMethodCall(u32 method, u32 argument, u32 nonshadow_argument, bool is_last_call); + /// CallMethod after the split's checks: CallMethodFast, or CallMethodSlow where it can't. + void DispatchMethod(Core::System& system, u32 method, u32 method_argument, bool is_last_call); + /// CallMethod with EDEN_NX2_GPU_PROFILE: DispatchMethod, counted and timed. + void ProfiledCallMethod(Core::System& system, u32 method, u32 method_argument, + bool is_last_call); /// CallMethod with every check, for the cases and tools CallMethodFast leaves out. void CallMethodSlow(Core::System& system, u32 method, u32 method_argument, bool is_last_call); void CallMethodFast(u32 method, u32 method_argument, bool is_last_call); @@ -3394,6 +3443,12 @@ class Maxwell3D final : public EngineInterface { // model). Read per engine, not at static initialization, which runs before the frontend // loads nx2_switches.txt. const bool trace_first_method_writes = std::getenv("EDEN_NX2_3D_METHODS") != nullptr; + // EDEN_NX2_DMA_READ_VERIFY=1: a macro DMA read the front makes without draining the split + // (FrontDmaRead) drains it anyway and has the caches write the words out, as every read + // did before, and the macro gets those words. The front counts the reads whose words + // differ from the ones it read first and logs the counts every 10 s, the first 16 reads + // with their words. The differences must stay 0. Read per engine, as the switch above. + const bool dma_read_verify = std::getenv("EDEN_NX2_DMA_READ_VERIFY") != nullptr; // EDEN_NX2_SKIP_SAME and EDEN_NX2_PERF_VERIFY, read per engine like the switch above. Every // method checks them, so they are members rather than statics with a guard on the hot path. const bool skip_same_value_writes = [] { @@ -3426,6 +3481,8 @@ class Maxwell3D final : public EngineInterface { void FrontRunMacro(Core::System& system, VideoCommon::GpuSplit& split, FrontState& state); bool FrontRunOnBack(Core::System& system, VideoCommon::GpuSplit& split, FrontState& state, const MacroEngine& engine, u32 start); + std::vector> FrontRunTheOldWay(VideoCommon::GpuSplit& split, + const FrontState& state, u32 start); u32 FrontRegister(VideoCommon::GpuSplit& split, FrontState& state, u32 method); bool FrontTakeOver(VideoCommon::GpuSplit& split, FrontState& state, u32 method); void FrontTrackInstanceLoop(FrontState& state, u32 method, u32 value); diff --git a/src/video_core/engines/maxwell_dma.cpp b/src/video_core/engines/maxwell_dma.cpp index 0b0541799b..c6b79a4c20 100644 --- a/src/video_core/engines/maxwell_dma.cpp +++ b/src/video_core/engines/maxwell_dma.cpp @@ -4,8 +4,12 @@ // SPDX-FileCopyrightText: Copyright 2018 yuzu Emulator Project // SPDX-License-Identifier: GPL-2.0-or-later +#include +#include #include +#include #include "common/algorithm.h" +#include "common/alignment.h" #include "common/assert.h" #include "common/logging.h" #include @@ -35,6 +39,79 @@ void MaxwellDMA::BindRasterizer(VideoCore::RasterizerInterface* rasterizer_) { rasterizer = rasterizer_; } +MaxwellDMA::LaunchWrites MaxwellDMA::WritesOfLaunch(std::span registers) { + const auto word = [registers](size_t offset) { return registers[offset / sizeof(u32)]; }; + const auto address = [&word](size_t offset) { + return static_cast(PackedGPUVAddr{word(offset), word(offset + sizeof(u32))}); + }; + // The power of two bytes a swizzling copy moves per element (as the copies compute it). + const auto bpp_shift = [](auto... values) { + return Common::FoldRight( + 4U, [](u32 x, u32 y) { return (std::min)(x, static_cast(std::countr_zero(y))); }, + static_cast(values)...); + }; + LaunchWrites writes{}; + const auto launch = std::bit_cast(word(offsetof(Regs, launch_dma))); + if (launch.semaphore_type == LaunchDMA::SemaphoreType::RELEASE_ONE_WORD_SEMAPHORE) { + writes.semaphore = {address(offsetof(Regs, semaphore)), sizeof(u32)}; + } else if (launch.semaphore_type == LaunchDMA::SemaphoreType::RELEASE_FOUR_WORD_SEMAPHORE) { + writes.semaphore = {address(offsetof(Regs, semaphore)), 4 * sizeof(u32)}; + } + const GPUVAddr in = address(offsetof(Regs, offset_in)); + const GPUVAddr out = address(offsetof(Regs, offset_out)); + const s32 pitch_out = static_cast(word(offsetof(Regs, pitch_out))); + const u32 length = word(offsetof(Regs, line_length_in)); + const u32 lines = word(offsetof(Regs, line_count)); + RemapConst remap; + std::memcpy(&remap, ®isters[offsetof(Regs, remap_const) / sizeof(u32)], sizeof(remap)); + DMA::Parameters dst; + std::memcpy(&dst, ®isters[offsetof(Regs, dst_params) / sizeof(u32)], sizeof(dst)); + DMA::Parameters src; + std::memcpy(&src, ®isters[offsetof(Regs, src_params) / sizeof(u32)], sizeof(src)); + const bool remapping = launch.remap_enable != 0; + const u32 base_bpp = !remapping ? 1U + : (remap.num_dst_components_minus_one + 1) * + (remap.component_size_minus_one + 1); + if (launch.multi_line_enable) { + const bool src_pitch = launch.src_memory_layout == LaunchDMA::MemoryLayout::PITCH; + const bool dst_pitch = launch.dst_memory_layout == LaunchDMA::MemoryLayout::PITCH; + if (!dst_pitch) { + // Swizzled into the destination surface (CopyBlockLinearToBlockLinear, + // CopyPitchToBlockLinear, or the texture cache's BufferToImage). + u32 shift = 0; + if (!remapping) { + shift = src_pitch ? bpp_shift(dst.width, length, dst.origin.x.Value(), out) + : bpp_shift(src.width, dst.width, length, src.origin.x.Value(), + dst.origin.x.Value(), in, out); + } + writes.destination = {out, CalculateSize(true, base_bpp << shift, dst.width >> shift, + dst.height, dst.depth, dst.block_size.height, + dst.block_size.depth)}; + } else if (!src_pitch) { + // Lines the pitch apart (CopyBlockLinearToPitch, or the cache's ImageToBuffer). + const u64 pitch = static_cast(std::abs(s64{pitch_out})); + writes.destination = {out, (std::max)(pitch, u64{length}) * lines}; + } else if (lines != 0 && length != 0) { + // Lines from the first to the last, whichever way the pitch goes. + const s64 last = s64{lines - 1} * pitch_out; + const u64 span = static_cast(last < 0 ? -last : last); + writes.destination = {last < 0 ? out - span : out, span + length}; + } + } else if (length != 0) { + if (remapping && remap.dst_x == RemapConst::Swizzle::CONST_A) { + // A fill: BufferClear counts words, the guest memory write takes length components. + writes.destination = {out, u64{length} * sizeof(u32)}; + } else { + // A copy; with a block linear page kind on either side, in 16-byte pieces swizzled + // within their 512-byte blocks. + const GPUVAddr begin = out & ~GPUVAddr{0x1ff}; + const GPUVAddr end = Common::AlignUp(out + length, GPUVAddr{0x200}); + writes.destination = {begin, end - begin}; + } + } + return writes; +} + void MaxwellDMA::ConsumeSinkImpl(Core::System& system) { for (auto [method, value] : method_sink) { regs.reg_array[method] = value; diff --git a/src/video_core/engines/maxwell_dma.h b/src/video_core/engines/maxwell_dma.h index 82ac274598..52b6ade13c 100644 --- a/src/video_core/engines/maxwell_dma.h +++ b/src/video_core/engines/maxwell_dma.h @@ -8,6 +8,7 @@ #include #include +#include #include #include "common/bit_field.h" @@ -244,6 +245,21 @@ class MaxwellDMA final : public EngineInterface { void BindRasterizer(VideoCore::RasterizerInterface* rasterizer); + /// The methods of the register file (CallMethod ignores the ones past it). + static constexpr u32 RegisterCount = 0x800; + /// Guest memory [address, address + size) a launch writes (size 0: none). + struct WriteRange { + GPUVAddr address{}; + u64 size{}; + }; + struct LaunchWrites { + WriteRange destination; ///< Every byte Launch() may write, whatever layout it picks + WriteRange semaphore; ///< ReleaseSemaphore's + }; + /// What a launch with register file `registers` writes (the split GPU thread's front keeps a + /// copy of the registers to place the copies the back makes). + [[nodiscard]] static LaunchWrites WritesOfLaunch(std::span registers); + explicit MaxwellDMA(MemoryManager& memory_manager_); ~MaxwellDMA() override; @@ -333,6 +349,7 @@ class MaxwellDMA final : public EngineInterface { #undef ASSERT_REG_POSITION static_assert(SemaphoreMethod * sizeof(u32) == offsetof(Regs, semaphore)); static_assert(LaunchDmaMethod * sizeof(u32) == offsetof(Regs, launch_dma)); + static_assert(RegisterCount == NUM_REGS); }; } // namespace Tegra::Engines diff --git a/src/video_core/engines/puller.cpp b/src/video_core/engines/puller.cpp index 8d8af23d86..02722c32ad 100644 --- a/src/video_core/engines/puller.cpp +++ b/src/video_core/engines/puller.cpp @@ -24,6 +24,7 @@ #include "video_core/gpu_split.h" #include "video_core/memory_manager.h" #include "video_core/nx2.h" +#include "video_core/nx2_drain_check.h" #include "video_core/nx2_fast_draw.h" #include "video_core/rasterizer_interface.h" @@ -34,40 +35,54 @@ namespace { constexpr u32 Gf100BindClassMask = 0xffff; constexpr u32 Gf100BindValidMask = 0x1f0000 | Gf100BindClassMask; -// Reads a semaphore word as a GPU acquire sees it, flushing a GPU-written copy out of the buffer -// cache first. A mirrored release the guest memory does not hold counts (Nx2MirrorShadow). +// Reads a semaphore word, flushing a GPU-written copy out of the buffer cache first. u32 ReadSemaphoreWord(DmaPusher& dma_pusher, GPUVAddr address) { - const auto device = dma_pusher.memory_manager.GpuToCpuAddress(address); - if (device && dma_pusher.rasterizer->MustFlushRegion(*device, sizeof(u32), - VideoCommon::CacheType::BufferCache)) { + if (const auto device = dma_pusher.memory_manager.GpuToCpuAddress(address); + device && dma_pusher.rasterizer->MustFlushRegion(*device, sizeof(u32), + VideoCommon::CacheType::BufferCache)) { VideoCommon::GpuSplit::Get().Run( [rasterizer = dma_pusher.rasterizer, device = *device] { rasterizer->FlushRegion(device, sizeof(u32), VideoCommon::CacheType::BufferCache); }, VideoCommon::DrainSite::SemaphoreFlush); } - const u32 guest = dma_pusher.memory_manager.Read(address); - if (!device) { - return guest; + const u32 word = dma_pusher.memory_manager.Read(address); + if (const auto device = dma_pusher.memory_manager.GpuToCpuAddress(address)) { + return Nx2MirrorShadow::Get().Read(*device, word).value_or(word); } - return Nx2MirrorShadow::Get().Read(*device, guest).value_or(guest); + return word; } // With the GPU thread split (gpu_split.h) the rasterizer belongs to the back thread: work that // only feeds it runs there in order; reads of what it wrote first wait for it (DrainBack). +// `writes`: what it writes of guest memory, for the front's reads (GpuSplit::Writes). template -void OnBack(const Func& func) { +void OnBack(const Func& func, + VideoCommon::GpuSplit::Writes writes = VideoCommon::GpuSplit::Writes::Anywhere) { if (auto& split = VideoCommon::GpuSplit::Get(); split.Active()) { - split.Defer(func); + split.Defer(func, writes); } else { func(); } } +// Fence work (ReleaseFences, fences and syncpoints signaled): downloads land in guest memory +// where the GPU wrote, nothing else. +constexpr auto FenceWrites = VideoCommon::GpuSplit::Writes::Caches; + void DrainBack() { VideoCommon::GpuSplit::Get().Drain(VideoCommon::DrainSite::Acquire); } +// EDEN_NX2_DRAIN_CHECK: whether an acquire (or a suspended one's poll) met its value before the +// drain as well as after it, and whether the GPU had released a semaphore at the address. +void NoteAcquireCheck(DmaPusher& dma_pusher, GPUVAddr address, bool poll, bool met_before, + bool met_after) { + const auto device = dma_pusher.memory_manager.GpuToCpuAddress(address); + const bool released = device && VideoCommon::Nx2DrainCheck::WasReleased(*device); + VideoCommon::Nx2DrainCheck::NoteAcquire(poll, released, met_before, met_after); +} + // The fence manager records into the scheduler and downloads the CPU's writes over GPU data: // the back's state, which a drained back still uses from its idle hook (it submits). void ReleaseFences(DmaPusher& dma_pusher) { @@ -76,25 +91,37 @@ void ReleaseFences(DmaPusher& dma_pusher) { VideoCommon::DrainSite::FenceRelease); } -// A release was just sent to the back: tell the front what it writes (CheckWithoutDrain). -void NoteRelease(DmaPusher& dma_pusher, GPUVAddr address, u32 payload, bool four_words) { +// A release with `properties` was just sent to the back (as Writes::Noted): tell the front what +// it writes (CheckWithoutDrain). +void NoteRelease(DmaPusher& dma_pusher, GPUVAddr address, u32 payload, + VideoCommon::QueryPropertiesFlags properties) { auto& split = VideoCommon::GpuSplit::Get(); if (!split.Active()) { return; } + const bool four_words = True(properties & VideoCommon::QueryPropertiesFlags::HasTimeout); + // A fence-flagged payload release lands as GpuSplit::FencedPayloadLanding says. + const auto landing = True(properties & VideoCommon::QueryPropertiesFlags::IsAFence) + ? VideoCommon::GpuSplit::FencedPayloadLanding() + : VideoCommon::GpuSplit::Landing::AtRun; + split.NoteWrite(dma_pusher.memory_manager, address, four_words ? 16 : sizeof(u32)); const auto device = dma_pusher.memory_manager.GpuToCpuAddress(address); if (!device) { return; } - split.NoteSemaphoreWrite(*device, payload, true); + split.NoteSemaphoreWrite(*device, payload, true, false, landing); + VideoCommon::Nx2AcquireStats::NoteRelease(*device, + VideoCommon::Nx2AcquireStats::Releaser::Puller); if (four_words) { // The payload as 64 bits (a timestamp follows). - split.NoteSemaphoreWrite(*device + 4, 0, true); + split.NoteSemaphoreWrite(*device + 4, 0, true, false, landing); } } -// Whether the back writes a payload release to memory when it runs it (CounterReport): below -// High GPU accuracy, with fences that run their operations at once (FenceManager::SignalFence). +// Whether the back writes a payload report without a fence flag to memory when it runs it +// (CounterReport): below High GPU accuracy, with fences that run their operations at once +// (FenceManager::SignalFence). Fence-flagged reports land once their fence ran +// (GpuSplit::Landing::Fenced). bool ReleasesLandAtOnce() { const bool delay_fence = Settings::IsGPUFenceBehaviorDefault() @@ -103,77 +130,210 @@ bool ReleasesLandAtOnce() { return !delay_fence && !Settings::IsGPULevelHigh(); } -enum class AcquireCheck { - Met, ///< By the word in memory - MetByQueued, ///< By a release sent to the word that the back has not run yet - NotMet, ///< Nothing sent to the word can meet it yet - Drain, ///< Drain the split and read the word as before +// Whether the front decides Switch 2 acquires from its view of the semaphore words (AskView) +// instead of draining the split: the back writes each release to memory as it runs it, or once +// its fence ran (GpuSplit::Landing), and the front sees every release it sends, its macros' +// included. +bool ViewApplies() { + const auto& split = VideoCommon::GpuSplit::Get(); + return split.Active() && split.FrontMacros() && ReleasesLandAtOnce(); +} + +// What the front can tell of an acquire without waiting for the back. +struct ViewAnswer { + enum class Kind { + Met, ///< By the word in memory + MetByQueued, ///< By the last release sent to the word, which the back has not run yet + NotMet, ///< Nothing sent to the word meets it yet + Unknown, ///< Memory tells once the back ran the work before `position` + Flush, ///< Shaders wrote the word: the buffer cache may hold it newer than memory + }; + Kind kind; + u64 position{}; }; -// Checks a greater-or-equal acquire without draining the split. The words Switch 2 syncs on only -// advance (NVN2 even releases QMD completions as maxima, KeplerCompute::ProcessNativeMethod), so -// a value that meets one now still does once the back ran the queued work. Only the last release -// the front sent to the word decides instead, when the back may not have run it yet. -AcquireCheck CheckWithoutDrain(DmaPusher& dma_pusher, const Puller::PendingAcquire& acquire) { - auto& split = VideoCommon::GpuSplit::Get(); - auto& stats = VideoCommon::g_acquire_stats; - if (!acquire.WaitsToReach()) { - VideoCommon::Count(stats.drain_operation); - return AcquireCheck::Drain; +ViewAnswer AskView(DmaPusher& dma_pusher, const Puller::PendingAcquire& acquire, u64 device) { + using Kind = ViewAnswer::Kind; + const auto& split = VideoCommon::GpuSplit::Get(); + const auto memory = [&] { return dma_pusher.memory_manager.Read(acquire.address); }; + // The words Switch 2 syncs on only advance (NVN2 even releases QMD completions as maxima, + // KeplerCompute::ProcessNativeMethod): memory that meets a greater-or-equal acquire now still + // does once the back ran the queued work, also where the buffer cache holds it newer. + if (Nx2MayBeGpuWritten(device, sizeof(u32)) && + dma_pusher.rasterizer->MustFlushRegion(device, sizeof(u32), + VideoCommon::CacheType::BufferCache)) { + return {acquire.WaitsToReach() && acquire.Matches(memory()) ? Kind::Met : Kind::Flush}; + } + // Another acquire holds only for the value the word has when the channel reaches it. Compute + // and copy work sent before it may still release the word without the front's knowing (QMD + // releases): a value that meets it stands once the back ran that work. + const bool settled = acquire.WaitsToReach() || split.Ran(acquire.writers_before); + // The last release the front sent to the word decides while it may not be in memory. + const auto queued = split.QueuedSemaphoreWriteAt(device); + if (queued && queued->fenced) { + // It lands once a fence ran, at no position the front can wait for: memory decides, as + // after a drain, and the landing latches the waiter or a poll finds the word. Writes sent + // to the word before it are in memory once the back ran it. A greater-or-equal acquire + // memory meets now meets it then too (the words only advance). + if (!split.Ran(queued->position) && !(queued->known && acquire.WaitsToReach())) { + return {Kind::Unknown, queued->position}; + } + } else if (queued) { + if (!queued->known) { + return {Kind::Unknown, queued->position}; + } + if (queued->at_least) { + // A maximum (QMD releases): the word holds the payload or more once the back ran it, + // which meets a greater-or-equal acquire of the payload or less. Else memory tells. + return acquire.WaitsToReach() && acquire.Matches(queued->payload) + ? ViewAnswer{Kind::MetByQueued} + : ViewAnswer{Kind::Unknown, queued->position}; + } + if (!acquire.Matches(queued->payload)) { + return {Kind::NotMet}; + } + if (settled || queued->position >= acquire.writers_before) { + return {Kind::MetByQueued}; + } + return {Kind::Unknown, acquire.writers_before}; } - if (!split.Active() || !split.FrontMacros() || !ReleasesLandAtOnce()) { - VideoCommon::Count(stats.drain_settings); - return AcquireCheck::Drain; + if (!acquire.Matches(memory())) { + return {Kind::NotMet}; } - const auto device = dma_pusher.memory_manager.GpuToCpuAddress(acquire.address); - // A word the buffer cache may hold newer than memory is read through the back. - if (!device || Nx2MayBeGpuWritten(*device, sizeof(u32))) { - VideoCommon::Count(stats.drain_word); - return AcquireCheck::Drain; - } - if (const auto queued = split.QueuedSemaphoreWriteAt(*device)) { - if (queued->known && acquire.Matches(queued->payload)) { - return AcquireCheck::MetByQueued; + return settled ? ViewAnswer{Kind::Met} : ViewAnswer{Kind::Unknown, acquire.writers_before}; +} + +// Has the back flush the word out of the buffer cache after the work sent before: memory holds +// it once the back ran the flush (PendingAcquire::flush_position). +void RequestFlush(DmaPusher& dma_pusher, Puller::PendingAcquire& acquire, u64 device) { + auto& split = VideoCommon::GpuSplit::Get(); + split.Defer( + [rasterizer = dma_pusher.rasterizer, device] { + rasterizer->FlushRegion(device, sizeof(u32), VideoCommon::CacheType::BufferCache); + }, + VideoCommon::GpuSplit::Writes::Caches); + acquire.flush_position = split.Position(); + acquire.flushed = true; +} + +// EDEN_NX2_DRAIN_CHECK: what the drain path decides instead, to verify the view (a release the +// back ran latches `waiter`, a suspended acquire's). +bool MetAfterDrain(DmaPusher& dma_pusher, const Puller::PendingAcquire& acquire, + const void* waiter) { + DrainBack(); + return acquire.Matches(ReadSemaphoreWord(dma_pusher, acquire.address)) || + (waiter != nullptr && Nx2SemaphoreWaiters::Get().Latched(waiter)); +} + +// TEST (EDEN_NX2_DRAIN_CHECK): whether a fence-flagged release to the word at `device` may still +// land. One that lands during the drain can change the word the view read, as it can on the +// drain path itself. +bool FencedWritePending(u64 device) { + const auto queued = VideoCommon::GpuSplit::Get().QueuedSemaphoreWriteAt(device); + return queued && queued->fenced; +} + +// TEST: checks a pass of the view against the drain path: whether the drained word meets it, and +// whether a fenced release that landed during the drain changed it. +bool PassStands(DmaPusher& dma_pusher, const Puller::PendingAcquire& acquire, const void* waiter, + u64 device, bool poll, VideoCommon::Nx2DrainCheck::ViewPass how) { + const bool fenced = FencedWritePending(device); + const bool met = MetAfterDrain(dma_pusher, acquire, waiter); + VideoCommon::Nx2DrainCheck::NoteViewPass(poll, how, met, + !met && fenced && !FencedWritePending(device)); + return met; +} + +// TEST: counts the view's decision on an acquire (EDEN_NX2_ACQUIRE_STATS) and checks it against +// the drain path (EDEN_NX2_DRAIN_CHECK): a pass the drained word doesn't meet suspends instead, +// a suspension the drained word meets passes. +ViewAnswer CheckAcquire(DmaPusher& dma_pusher, const Puller::PendingAcquire& acquire, + ViewAnswer answer, bool waited, u64 device) { + using Kind = ViewAnswer::Kind; + namespace Check = VideoCommon::Nx2DrainCheck; + namespace Stats = VideoCommon::Nx2AcquireStats; + const bool pass = answer.kind == Kind::Met || answer.kind == Kind::MetByQueued; + const auto how = waited ? Check::ViewPass::Waited + : answer.kind == Kind::MetByQueued ? Check::ViewPass::Queued + : Check::ViewPass::Memory; + if (Stats::g_enabled) { + const auto decision = !pass ? Stats::Decision::Suspended + : how == Check::ViewPass::Waited ? Stats::Decision::Waited + : how == Check::ViewPass::Queued ? Stats::Decision::Queued + : Stats::Decision::Memory; + Stats::NoteAcquire(acquire.operation, decision, device, waited, + Nx2MayBeGpuWritten(device, sizeof(u32)), answer.kind == Kind::Flush); + } + if (Check::g_enabled) { + if (pass) { + if (!PassStands(dma_pusher, acquire, nullptr, device, false, how)) { + return {Kind::NotMet}; + } + } else { + const bool met = MetAfterDrain(dma_pusher, acquire, nullptr); + Check::NoteViewWait(false, met); + if (met) { + return {Kind::Met}; + } } - VideoCommon::Count(stats.drain_queued); - return AcquireCheck::Drain; } - return acquire.Matches(dma_pusher.memory_manager.Read(acquire.address)) - ? AcquireCheck::Met - : AcquireCheck::NotMet; + return answer; } -// EDEN_NX2_ACQUIRE_VERIFY=1: drains anyway and counts where the word after the drain disagrees -// with what CheckWithoutDrain decided. -void VerifyWithoutDrain(DmaPusher& dma_pusher, const Puller::PendingAcquire& acquire, - AcquireCheck decided) { - if (!VideoCommon::g_acquire_verify) { - return; +// TEST: whether the view's resume of a poll stands (EDEN_NX2_DRAIN_CHECK). +bool PollPassStands(DmaPusher& dma_pusher, const Puller::PendingAcquire& acquire, + const void* waiter, u64 device, VideoCommon::Nx2DrainCheck::ViewPass how) { + if (!VideoCommon::Nx2DrainCheck::g_enabled) { + return true; } - DrainBack(); - const u32 word = ReadSemaphoreWord(dma_pusher, acquire.address); - auto& stats = VideoCommon::g_acquire_stats; - if (acquire.Matches(word)) { - if (decided == AcquireCheck::NotMet) { - VideoCommon::Count(stats.late); - } + return PassStands(dma_pusher, acquire, waiter, device, true, how); +} + +// TEST: a suspended acquire resumes (EDEN_NX2_ACQUIRE_STATS). +void NoteResumed(const void* waiter, const Puller::PendingAcquire& acquire, + VideoCommon::Nx2AcquireStats::Resume how) { + namespace Stats = VideoCommon::Nx2AcquireStats; + if (!Stats::g_enabled) { return; } - if (decided == AcquireCheck::MetByQueued) { - // A fence-flagged release writes memory once the host GPU passed it. - VideoCommon::Count(stats.unlanded); + const auto source = how == Stats::Resume::Latch ? Nx2SemaphoreWaiters::Get().LatchSource(waiter) + : Stats::Source::Unknown; + Stats::NoteResume(how, source, acquire.flushed, + std::chrono::steady_clock::now() - acquire.since); +} + +// The fence work an acquire that still waits after the drain sends (ReleaseFences), sent once the +// back ran everything before the acquire: the release that work runs then counts as it did (the +// waiter is registered), and acquires the queued work releases send none. +void MaybeReleaseFences(DmaPusher& dma_pusher, Puller::PendingAcquire& acquire) { + if (acquire.fences_released || !VideoCommon::GpuSplit::Get().Ran(acquire.sent_at)) { return; } - if (decided != AcquireCheck::Met) { + acquire.fences_released = true; + OnBack([rasterizer = dma_pusher.rasterizer] { rasterizer->ReleaseFences(); }, FenceWrites); +} + +// Logs once an acquire that waits for more than 2 s. `current`: the word as read after a drain, +// or nothing to read it from memory. +void ReportLongWait(DmaPusher& dma_pusher, Puller::PendingAcquire& acquire, + std::optional current) { + if (acquire.reported || + std::chrono::steady_clock::now() - acquire.since <= std::chrono::seconds{2}) { return; } - VideoCommon::Count(stats.mismatches); - if (static std::atomic logged{}; logged.fetch_add(1) < 16) { - LOG_WARNING(HW_GPU, - "Acquire verify: channel {} passed semaphore {:#x} (>= {:#x}) without a " - "drain, the word is {:#x} after it", - dma_pusher.channel_state.bind_id, acquire.address, acquire.value, word); - } + acquire.reported = true; + const auto device = dma_pusher.memory_manager.GpuToCpuAddress(acquire.address); + const GPUVAddr words = acquire.address & ~GPUVAddr{0xf}; + LOG_WARNING( + HW_GPU, + "Channel {} waits on semaphore {:#x} (op {}, {:#x}, now {:#x}) for 2 s; device " + "{:#x}, buffer cache holds GPU data: {}, words {:08x} {:08x} {:08x} {:08x}", + dma_pusher.channel_state.bind_id, acquire.address, acquire.operation, acquire.value, + current.value_or(dma_pusher.memory_manager.Read(acquire.address)), device.value_or(0), + device && dma_pusher.rasterizer->MustFlushRegion(*device, 16, VideoCommon::CacheType::All), + dma_pusher.memory_manager.Read(words), dma_pusher.memory_manager.Read(words + 4), + dma_pusher.memory_manager.Read(words + 8), + dma_pusher.memory_manager.Read(words + 12)); } } // Anonymous namespace @@ -196,48 +356,26 @@ bool Puller::PendingAcquire::WaitsToReach() const { } bool Puller::CanResumeAcquire(DmaPusher& dma_pusher) { - if (!pending_acquire) { - return true; - } - auto& stats = VideoCommon::g_acquire_stats; - switch (const AcquireCheck check = CheckWithoutDrain(dma_pusher, *pending_acquire); check) { - case AcquireCheck::MetByQueued: - // What the front reads after the acquire waits for the work sent before the release. - VideoCommon::GpuSplit::Get().NoteSync(); - VideoCommon::Count(stats.by_queued); - [[fallthrough]]; - case AcquireCheck::Met: - VideoCommon::Count(stats.passed); - VerifyWithoutDrain(dma_pusher, *pending_acquire, check); + if (!pending_acquire || pending_acquire->met) { return true; - case AcquireCheck::NotMet: { - // A release the back ran latches the waiter. Otherwise poll again later, without waiting - // for the back: until it runs one, nothing sent to the word meets it. Work only a drain - // finishes (the back's idle hook submits what the host GPU writes) still lands: a waiting - // acquire drains at most once per 1 ms (PantACRO4life's valve, b837e742c4), and the drain - // reports waits over 2 s. - if (Nx2SemaphoreWaiters::Get().Latched(this)) { - VideoCommon::Count(stats.latched); - return true; - } - const auto now = std::chrono::steady_clock::now(); - if (now - pending_acquire->last_drain < std::chrono::milliseconds{1}) { - VideoCommon::Count(stats.polls); - VerifyWithoutDrain(dma_pusher, *pending_acquire, check); - return false; - } - pending_acquire->last_drain = now; - VideoCommon::Count(stats.valve_drains); - break; } - case AcquireCheck::Drain: - break; + if (const auto device = dma_pusher.memory_manager.GpuToCpuAddress(pending_acquire->address); + device && ViewApplies()) { + pending_acquire->met = PollWithoutDrain(dma_pusher, *device); + return pending_acquire->met; } - VideoCommon::Count(stats.drained); + const bool check = VideoCommon::Nx2DrainCheck::g_enabled; + const bool met_before = + check && + pending_acquire->Matches(dma_pusher.memory_manager.Read(pending_acquire->address)); // The release may still be queued for the split GPU thread's back half (and the caches // ReadSemaphoreWord asks are its state). DrainBack(); const u32 current = ReadSemaphoreWord(dma_pusher, pending_acquire->address); + if (check) [[unlikely]] { + NoteAcquireCheck(dma_pusher, pending_acquire->address, true, met_before, + pending_acquire->Matches(current)); + } if (Nx2SemaphoreWaiters::Get().Latched(this)) { if (static std::atomic rescued{}; !pending_acquire->Matches(current) && rescued.fetch_add(1) < 64) { @@ -252,7 +390,9 @@ bool Puller::CanResumeAcquire(DmaPusher& dma_pusher) { } // With delayed fence operations (Balanced or Accurate fences, accurate GPU), a release // queued after the acquire released the fences waits in the caches for the next fence. With - // every channel waiting none comes: release them again now and then while waiting. + // every channel waiting none comes: release them again now and then while waiting. (The + // semaphore view, PollWithoutDrain, applies only while releases without a fence flag land at + // once: never with delayed fence operations. Fence-flagged ones signal their own fence.) const bool delayed = Settings::IsGPUFenceBehaviorDefault() ? Settings::IsGPULevelHigh() : Settings::IsGPUFenceBehaviorBalanced() || @@ -266,20 +406,66 @@ bool Puller::CanResumeAcquire(DmaPusher& dma_pusher) { return true; } } - if (!pending_acquire->reported && - std::chrono::steady_clock::now() - pending_acquire->since > std::chrono::seconds{2}) { - pending_acquire->reported = true; - const auto device = dma_pusher.memory_manager.GpuToCpuAddress(pending_acquire->address); - LOG_WARNING(HW_GPU, "Channel {} waits on semaphore {:#x} (op {}, {:#x}, now {:#x}) for 2 s; " - "device {:#x}, buffer cache holds GPU data: {}, words {:08x} {:08x} {:08x} {:08x}", - dma_pusher.channel_state.bind_id, pending_acquire->address, - pending_acquire->operation, pending_acquire->value, current, device.value_or(0), - device && dma_pusher.rasterizer->MustFlushRegion(*device, 16, VideoCommon::CacheType::All), - dma_pusher.memory_manager.Read(pending_acquire->address & ~GPUVAddr{0xf}), - dma_pusher.memory_manager.Read((pending_acquire->address & ~GPUVAddr{0xf}) + 4), - dma_pusher.memory_manager.Read((pending_acquire->address & ~GPUVAddr{0xf}) + 8), - dma_pusher.memory_manager.Read((pending_acquire->address & ~GPUVAddr{0xf}) + 12)); + ReportLongWait(dma_pusher, *pending_acquire, current); + return false; +} + +bool Puller::PollWithoutDrain(DmaPusher& dma_pusher, u64 device) { + using Kind = ViewAnswer::Kind; + using Resume = VideoCommon::Nx2AcquireStats::Resume; + using ViewPass = VideoCommon::Nx2DrainCheck::ViewPass; + PendingAcquire& acquire = *pending_acquire; + auto& split = VideoCommon::GpuSplit::Get(); + // A release the back ran latches the waiter, even if a later one changed the word again. + if (Nx2SemaphoreWaiters::Get().Latched(this)) { + NoteResumed(this, acquire, Resume::Latch); + return true; } + if (acquire.flush_position && split.Ran(*acquire.flush_position)) { + acquire.flush_position.reset(); + } + // TEST: a resume the drained word contradicts was checked already. + bool checked = false; + if (!acquire.flush_position) { + const ViewAnswer answer = AskView(dma_pusher, acquire, device); + switch (answer.kind) { + case Kind::Met: + if (!PollPassStands(dma_pusher, acquire, this, device, ViewPass::Memory)) { + checked = true; + break; + } + NoteResumed(this, acquire, Resume::Memory); + return true; + case Kind::MetByQueued: + if (!PollPassStands(dma_pusher, acquire, this, device, ViewPass::Queued)) { + checked = true; + break; + } + // What the front reads after the acquire waits for the work sent before the release. + split.NoteSync(); + NoteResumed(this, acquire, Resume::Queued); + return true; + case Kind::Flush: + RequestFlush(dma_pusher, acquire, device); + break; + case Kind::NotMet: + case Kind::Unknown: + // Poll again later, without waiting for the back: until it runs a release that meets + // the acquire (which latches the waiter) or the work before `position`, nothing the + // front sees meets it. + break; + } + } + if (VideoCommon::Nx2DrainCheck::g_enabled && !checked) [[unlikely]] { + const bool met = MetAfterDrain(dma_pusher, acquire, this); + VideoCommon::Nx2DrainCheck::NoteViewWait(true, met); + if (met) { + NoteResumed(this, acquire, Resume::Drain); + return true; + } + } + MaybeReleaseFences(dma_pusher, acquire); + ReportLongWait(dma_pusher, acquire, std::nullopt); return false; } @@ -358,13 +544,14 @@ void Puller::ProcessFenceActionMethod(DmaPusher& dma_pusher) { case Puller::FenceOperation::Acquire: // UNIMPLEMENTED_MSG("Channel Scheduling pending."); // WaitFence(regs.fence_action.syncpoint_id, regs.fence_value); - OnBack([rasterizer = dma_pusher.rasterizer] { rasterizer->ReleaseFences(); }); + OnBack([rasterizer = dma_pusher.rasterizer] { rasterizer->ReleaseFences(); }, FenceWrites); VideoCommon::GpuSplit::Get().NoteSync(); break; case Puller::FenceOperation::Increment: - OnBack([rasterizer = dma_pusher.rasterizer, id = static_cast(regs.fence_action.syncpoint_id.Value())] { - rasterizer->SignalSyncPoint(id); - }); + OnBack([rasterizer = dma_pusher.rasterizer, + id = static_cast( + regs.fence_action.syncpoint_id.Value())] { rasterizer->SignalSyncPoint(id); }, + FenceWrites); break; default: UNIMPLEMENTED_MSG("Unimplemented operation {}", regs.fence_action.op.Value()); @@ -396,19 +583,25 @@ void Puller::ProcessSemaphoreTriggerMethod(DmaPusher& dma_pusher) { const auto properties = g_nx2_ampere && (regs.semaphore_trigger & (1U << 24)) != 0 ? VideoCommon::QueryPropertiesFlags::IsAFence : VideoCommon::QueryPropertiesFlags::HasTimeout; - const GPUVAddr mirror = g_nx2_ampere ? Nx2SemaphoreMirror(sequence_address) : 0; + // The full front view values releases at their original addresses. A second release + // would change its decisions; mirroring belongs only to the non-view path. + const GPUVAddr mirror = + g_nx2_ampere && !ViewApplies() ? Nx2SemaphoreMirror(sequence_address) : 0; + OnBack( + [rasterizer = dma_pusher.rasterizer, sequence_address, mirror, properties, payload] { + rasterizer->Query(sequence_address, VideoCommon::QueryType::Payload, properties, + payload, 0); + if (mirror != 0) { + rasterizer->Query(mirror, VideoCommon::QueryType::Payload, + properties | VideoCommon::QueryPropertiesFlags::Nx2Mirror, + payload, 0); + } + }, + VideoCommon::GpuSplit::Writes::Noted); + NoteRelease(dma_pusher, sequence_address, payload, properties); if (mirror != 0) { - LOG_DEBUG(HW_GPU, "Switch 2 semaphore {:#x}={:#x} mirrored to {:#x}", sequence_address, payload, mirror); + NoteRelease(dma_pusher, mirror, payload, properties); } - OnBack([rasterizer = dma_pusher.rasterizer, sequence_address, mirror, properties, payload] { - rasterizer->Query(sequence_address, VideoCommon::QueryType::Payload, properties, payload, 0); - if (mirror != 0) { - rasterizer->Query(mirror, VideoCommon::QueryType::Payload, - properties | VideoCommon::QueryPropertiesFlags::Nx2Mirror, payload, 0); - } - }); - NoteRelease(dma_pusher, sequence_address, payload, - properties == VideoCommon::QueryPropertiesFlags::HasTimeout); } else if (g_nx2_ampere) { // Switch 2 waits on semaphores the CPU (or another channel) releases, e.g. once it has // written the commands this channel runs next: suspend the channel until the value is @@ -419,7 +612,6 @@ void Puller::ProcessSemaphoreTriggerMethod(DmaPusher& dma_pusher) { .operation = static_cast(op), .since = std::chrono::steady_clock::now(), .last_fences = std::chrono::steady_clock::now(), - .last_drain = std::chrono::steady_clock::now(), .reported = false, }; if (op != GpuSemaphoreOperation::AcquireEqual && op != GpuSemaphoreOperation::AcquireGequal && @@ -428,34 +620,30 @@ void Puller::ProcessSemaphoreTriggerMethod(DmaPusher& dma_pusher) { return; } VideoCommon::Nx2Profile::Add(VideoCommon::Nx2Profile::Counter::AcquireChecks, 1); - auto& stats = VideoCommon::g_acquire_stats; - if (const AcquireCheck without_drain = CheckWithoutDrain(dma_pusher, acquire); - without_drain == AcquireCheck::Met || without_drain == AcquireCheck::MetByQueued) { - if (without_drain == AcquireCheck::MetByQueued) { - // What the front reads after the acquire waits for the work sent before the - // release (GpuSplit::NoteSync). - VideoCommon::GpuSplit::Get().NoteSync(); - VideoCommon::Count(stats.by_queued); - } - VideoCommon::Count(stats.passed); - VerifyWithoutDrain(dma_pusher, acquire, without_drain); + if (AcquireWithoutDrain(dma_pusher, acquire)) { return; } - VideoCommon::Count(stats.drained); + if (VideoCommon::Nx2AcquireStats::g_enabled) [[unlikely]] { + VideoCommon::Nx2AcquireStats::NoteAcquire( + acquire.operation, VideoCommon::Nx2AcquireStats::Decision::Drained, 0, false, false, + false); + } + const bool check = VideoCommon::Nx2DrainCheck::g_enabled; + const bool met_before = + check && acquire.Matches(dma_pusher.memory_manager.Read(acquire.address)); DrainBack(); - if (acquire.Matches(ReadSemaphoreWord(dma_pusher, acquire.address))) { + const bool met = acquire.Matches(ReadSemaphoreWord(dma_pusher, acquire.address)); + if (check) [[unlikely]] { + NoteAcquireCheck(dma_pusher, acquire.address, false, met_before, met); + } + if (met) { return; } // Registered before the fences run, so a release among them that meets the value counts // even if a later one overwrites it. auto& waiters = Nx2SemaphoreWaiters::Get(); - if (const auto device = dma_pusher.memory_manager.GpuToCpuAddress(acquire.address); - device && !waiters.Add(this, *device, acquire.value, acquire.operation)) { - if (static std::atomic full{}; full.fetch_add(1) < 8) { - LOG_ERROR(HW_GPU, "Channel {} waits on semaphore {:#x} unlatched: {} channels wait", - dma_pusher.channel_state.bind_id, acquire.address, - Nx2SemaphoreWaiters::MaxWaiters); - } + if (const auto device = dma_pusher.memory_manager.GpuToCpuAddress(acquire.address)) { + waiters.Add(this, *device, acquire.value, acquire.operation); } ReleaseFences(dma_pusher); if (!waiters.Latched(this) && !acquire.Matches(ReadSemaphoreWord(dma_pusher, acquire.address))) { @@ -497,14 +685,63 @@ void Puller::ProcessSemaphoreTriggerMethod(DmaPusher& dma_pusher) { } } +bool Puller::AcquireWithoutDrain(DmaPusher& dma_pusher, PendingAcquire& acquire) { + using Kind = ViewAnswer::Kind; + const auto device = dma_pusher.memory_manager.GpuToCpuAddress(acquire.address); + if (!device || !ViewApplies()) { + return false; + } + auto& split = VideoCommon::GpuSplit::Get(); + acquire.writers_before = split.LastWriterPosition(); + acquire.sent_at = split.Position(); + ViewAnswer answer = AskView(dma_pusher, acquire, *device); + // The value is in memory once the back ran that work, before what was sent after it. A + // release the front can't value, then the writers before the acquire: two waits at most. + u32 waits = 0; + for (; answer.kind == Kind::Unknown && waits < 2; ++waits) { + split.WaitForPosition(answer.position, VideoCommon::DrainSite::Acquire); + answer = AskView(dma_pusher, acquire, *device); + } + if (VideoCommon::Nx2DrainCheck::g_enabled || VideoCommon::Nx2AcquireStats::g_enabled) + [[unlikely]] { + answer = CheckAcquire(dma_pusher, acquire, answer, waits != 0, *device); + } + switch (answer.kind) { + case Kind::Met: + return true; + case Kind::MetByQueued: + // What the front reads after the acquire waits for the work sent before the release + // (GpuSplit::NoteSync). + split.NoteSync(); + return true; + case Kind::Flush: + RequestFlush(dma_pusher, acquire, *device); + break; + case Kind::NotMet: + case Kind::Unknown: + break; + } + // Suspended: a release the back runs latches the waiter, registered before any fence work, + // and the polls see the others in memory or in the view (PollWithoutDrain). + Nx2SemaphoreWaiters::Get().Add(this, *device, acquire.value, acquire.operation); + MaybeReleaseFences(dma_pusher, acquire); + pending_acquire = acquire; + dma_pusher.suspensions.fetch_add(1, std::memory_order_release); + VideoCommon::Nx2Profile::Add(VideoCommon::Nx2Profile::Counter::AcquireWaits, 1); + return true; +} + void Puller::ProcessSemaphoreRelease(DmaPusher& dma_pusher) { const GPUVAddr sequence_address{regs.semaphore_address.SemaphoreAddress()}; const u32 payload = regs.semaphore_release; - OnBack([rasterizer = dma_pusher.rasterizer, sequence_address, payload] { - rasterizer->Query(sequence_address, VideoCommon::QueryType::Payload, - VideoCommon::QueryPropertiesFlags::IsAFence, payload, 0); - }); - NoteRelease(dma_pusher, sequence_address, payload, false); + OnBack( + [rasterizer = dma_pusher.rasterizer, sequence_address, payload] { + rasterizer->Query(sequence_address, VideoCommon::QueryType::Payload, + VideoCommon::QueryPropertiesFlags::IsAFence, payload, 0); + }, + VideoCommon::GpuSplit::Writes::Noted); + NoteRelease(dma_pusher, sequence_address, payload, + VideoCommon::QueryPropertiesFlags::IsAFence); } void Puller::ProcessSemaphoreAcquire(DmaPusher& dma_pusher) { @@ -532,32 +769,53 @@ void Puller::ProcessSemaphoreExecute(DmaPusher& dma_pusher, u32 argument) { const u32 payload = regs.reg_array[0x19]; const u32 operation = argument & 7; if (operation == 1) { // RELEASE - OnBack([rasterizer = dma_pusher.rasterizer, address, payload] { - rasterizer->Query(address, VideoCommon::QueryType::Payload, - VideoCommon::QueryPropertiesFlags::IsAFence, payload, 0); - }); - NoteRelease(dma_pusher, address, payload, false); + OnBack( + [rasterizer = dma_pusher.rasterizer, address, payload] { + rasterizer->Query(address, VideoCommon::QueryType::Payload, + VideoCommon::QueryPropertiesFlags::IsAFence, payload, 0); + }, + VideoCommon::GpuSplit::Writes::Noted); + NoteRelease(dma_pusher, address, payload, VideoCommon::QueryPropertiesFlags::IsAFence); return; } - // Acquires: check once after flushing pending fences; never block the GPU thread on them. - DrainBack(); - VideoCommon::Nx2FastDraw::Break(VideoCommon::Nx2FastDraw::Reason::Fence); - ReleaseFences(dma_pusher); - u32 word = dma_pusher.memory_manager.Read(address); - if (const auto device = dma_pusher.memory_manager.GpuToCpuAddress(address)) { - word = Nx2MirrorShadow::Get().Read(*device, word).value_or(word); - } - const bool ready = operation == 0 ? word == payload - : operation == 2 ? word >= payload - : operation == 3 ? static_cast(word - payload) >= 0 - : operation == 4 ? (word & payload) != 0 - : operation == 5 ? (word & payload) == 0 - : true; - if (!ready) { + // Acquires: never block the GPU thread on them; they are checked for the warning below only. + const auto ready = [operation, payload](u32 value) { + return operation == 0 ? value == payload + : operation == 2 ? value >= payload + : operation == 3 ? static_cast(value - payload) >= 0 + : operation == 4 ? (value & payload) != 0 + : operation == 5 ? (value & payload) == 0 + : true; + }; + std::optional word; + if (const auto device = dma_pusher.memory_manager.GpuToCpuAddress(address); + device && ViewApplies()) { + // The last release sent to the word tells while the back hasn't run it. What the front + // reads after the acquire waits for the work sent before it, as after a drain. + auto& split = VideoCommon::GpuSplit::Get(); + split.NoteSync(); + OnBack([rasterizer = dma_pusher.rasterizer] { rasterizer->ReleaseFences(); }, FenceWrites); + if (const auto queued = split.QueuedSemaphoreWriteAt(*device)) { + if (queued->known && !queued->at_least) { + word = queued->payload; + } + } else { + word = dma_pusher.memory_manager.Read(address); + } + } else { + DrainBack(); + VideoCommon::Nx2FastDraw::Break(VideoCommon::Nx2FastDraw::Reason::Fence); + ReleaseFences(dma_pusher); + word = dma_pusher.memory_manager.Read(address); + if (const auto device = dma_pusher.memory_manager.GpuToCpuAddress(address)) { + word = Nx2MirrorShadow::Get().Read(*device, *word).value_or(*word); + } + } + if (word && !ready(*word)) { static std::atomic reported{}; if (reported.fetch_add(1) < 8) { LOG_WARNING(HW_GPU, "SEM_EXECUTE op {} at {:#x}: {:#x} vs {:#x} not met, continuing", - operation, address, word, payload); + operation, address, *word, payload); } } } @@ -585,13 +843,15 @@ void Puller::CallPullerMethod(DmaPusher& dma_pusher, const MethodCall& method_ca case BufferMethods::WrcacheFlush: break; case BufferMethods::RefCnt: - OnBack([rasterizer = dma_pusher.rasterizer] { rasterizer->SignalReference(); }); + OnBack([rasterizer = dma_pusher.rasterizer] { rasterizer->SignalReference(); }, + FenceWrites); break; case BufferMethods::SyncpointOperation: ProcessFenceActionMethod(dma_pusher); break; case BufferMethods::WaitForIdle: - OnBack([rasterizer = dma_pusher.rasterizer] { rasterizer->WaitForIdle(); }); + OnBack([rasterizer = dma_pusher.rasterizer] { rasterizer->WaitForIdle(); }, + VideoCommon::GpuSplit::Writes::Caches); VideoCommon::GpuSplit::Get().NoteSync(); break; case BufferMethods::SemaphoreOperation: { @@ -608,7 +868,8 @@ void Puller::CallPullerMethod(DmaPusher& dma_pusher, const MethodCall& method_ca } case BufferMethods::MemOpB: { // Implement this better. - OnBack([rasterizer = dma_pusher.rasterizer] { rasterizer->InvalidateGPUCache(); }); + OnBack([rasterizer = dma_pusher.rasterizer] { rasterizer->InvalidateGPUCache(); }, + VideoCommon::GpuSplit::Writes::Caches); break; } case BufferMethods::MemOpC: diff --git a/src/video_core/engines/puller.h b/src/video_core/engines/puller.h index 8a464f7b31..5f6d82f80c 100644 --- a/src/video_core/engines/puller.h +++ b/src/video_core/engines/puller.h @@ -79,9 +79,22 @@ class Puller final { u32 operation; std::chrono::steady_clock::time_point since; std::chrono::steady_clock::time_point last_fences; ///< When it last released fences - /// When an unmet poll last drained (CanResumeAcquire) - std::chrono::steady_clock::time_point last_drain; - bool reported; ///< The 2 s warning was logged + bool reported; + /// GpuSplit::LastWriterPosition when the acquire was sent: compute and copy work before + /// it may release the word without the front's knowing (QMD releases). + u64 writers_before{}; + /// A flush of the word out of the buffer cache, sent to the split's back: until the back + /// ran it, memory may hold an older value than the cache. + std::optional flush_position{}; + /// GpuSplit::Position when the acquire was sent, and whether the fence work was sent + /// since: once the back ran everything before the acquire and it still waits, as after + /// the drain this replaces. + u64 sent_at{}; + bool fences_released{}; + /// CanResumeAcquire found it met: the channel resumes when it runs next, without + /// asking again (Control::Scheduler asks before it runs a suspended channel). + bool met{}; + bool flushed{}; ///< TEST (EDEN_NX2_ACQUIRE_STATS): a flush was sent for it [[nodiscard]] bool Matches(u32 current) const; /// Whether it waits for the word to reach `value` (greater or equal). @@ -151,6 +164,11 @@ class Puller final { void ProcessSemaphoreRelease(DmaPusher& dma_pusher); void ProcessSemaphoreTriggerMethod(DmaPusher& dma_pusher); void ProcessSemaphoreExecute(DmaPusher& dma_pusher, u32 argument); + /// Decides a Switch 2 acquire from the split's view of the semaphore words: passes it, or + /// suspends the channel on it. False when the view doesn't apply (drain as before). + bool AcquireWithoutDrain(DmaPusher& dma_pusher, PendingAcquire& acquire); + /// CanResumeAcquire for an acquire the view decides, at `device`. + bool PollWithoutDrain(DmaPusher& dma_pusher, u64 device); [[nodiscard]] bool ExecuteMethodOnEngine(DmaPusher& dma_pusher, u32 method); std::optional pending_acquire; diff --git a/src/video_core/fence_manager.h b/src/video_core/fence_manager.h index d7523db85f..48914d5e83 100644 --- a/src/video_core/fence_manager.h +++ b/src/video_core/fence_manager.h @@ -129,9 +129,6 @@ class FenceManager { func(); } fences.push(std::move(new_fence)); - if (should_flush) { - rasterizer.FlushCommands(); - } if constexpr (can_async_check) { guard.unlock(); cv.notify_all(); diff --git a/src/video_core/front_qmd.h b/src/video_core/front_qmd.h new file mode 100644 index 0000000000..3ecba92758 --- /dev/null +++ b/src/video_core/front_qmd.h @@ -0,0 +1,655 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common/common_types.h" +#include "video_core/gpu_split.h" +#include "video_core/native_qmd.h" +#include "video_core/nx2_drain_check.h" + +namespace VideoCommon { + +/// The QMD scheduling of every channel (Tegra::NativeQmdScheduler), run on the split GPU thread's +/// front as its back will run it, on a table of its own: what the launches release is known when +/// they are sent, for the front's view of the semaphore words (GpuSplit::NoteSemaphoreWrite). +/// +/// The table is the back's from the start: an engine's front takes it over with the back drained +/// when it takes the engine's macros over (TakeOver). It stays the back's as long as the front +/// reads the descriptors the back will read. The front reads one when it sends a signal, the back +/// when it runs it, both from guest memory without the GPU caches. The front takes a descriptor: +/// - from the signal's own inline QMD, which the back writes right before; +/// - from memory, once the back ran every write sent there (GpuSplit::MemoryWritesTarget: the +/// writes placed in its granule and the unplaced ones), on pages GPU work never wrote +/// (Nx2MayBeGpuWritten); +/// - from the inline QMD a LOAD_INLINE_QMD_DATA still queued writes there, when that is the last +/// write sent there (NoteInlineWrite). +/// No change of the GPU's mappings may be queued either (GpuSplit::LastMappingChange). Memory holds +/// what the back finds there only while the back has not run the signal's entry yet: the front +/// sends and values it under a GpuSplit::PublishHold. +/// +/// Where the back has yet to run a write the front can't read past (into a QMD the signal reads, +/// or anywhere, or to the mappings), the signal waits, with what it changed in the table taken +/// back (NativeQmdScheduler::BeginAttempt), and every signal sent after it waits behind it: the +/// table stays the back's as of the signals valued. The front values the waiting ones in order +/// once the back ran what the first waits for (CatchUp). Late, the back may have run the signal +/// and what followed it, so memory holds what the back read only where nothing sent after the +/// signal writes the QMD, and no release of the waiting signals writes a QMD read late. Their +/// releases are placed at their signals' positions, not noted. Where the front can't read what +/// the back read at all (a QMD on a GPU-written page, written after the signal, released into, +/// or the mappings changed after it), it loses the table: it stops valuing, and takes the back's +/// table over the next time the back has run everything sent, between commands or at a QMD +/// method (MaybeTakeOver). +/// +/// `Memory`, the channel's GPU address space as the back reads it, has: +/// - bool IsMapped(GPUVAddr, u64), void Read(GPUVAddr, void*, u64), u32 ReadWord(GPUVAddr) (as +/// MemoryManager::Read), bool InSyncpointShim(GPUVAddr, u64); +/// - std::optional Device(GPUVAddr), bool GpuWritten(u64 device, u64 size); +/// - u64 WritesTarget(const GpuSplit&, GPUVAddr, u64), void NoteWrite(GpuSplit&, GPUVAddr, u64), +/// void NoteWriteAt(GpuSplit&, GPUVAddr, u64 size, u64 position); +/// - const void* Id(): the address space's, the same for every Memory of it. +class FrontQmdView { +public: + using Descriptor = Tegra::NativeQmdScheduler::Descriptor; + + /// The view the compute engines of every channel feed: they share the back's table. + static FrontQmdView& Get() { + static FrontQmdView view; + return view; + } + + /// The back ran everything sent and runs nothing (the split drained, or it waits for work): + /// takes over the table `back` shares with the other engines. `where`: TEST. + void TakeOver(const Tegra::NativeQmdScheduler& back, Nx2AcquireStats::QmdTakeOver where) { + const auto start = std::chrono::steady_clock::now(); // TEST + scheduler.CopyStates(back); + synced = true; + Nx2AcquireStats::NoteQmdTakeOverDropped(waiting.size()); // TEST + ClearWaiting(); + Nx2AcquireStats::NoteQmdSync(where, std::chrono::steady_clock::now() - start); + } + + /// The front, between commands and before QMD methods: takes the back's table over if the + /// view lost it or signals wait, and the back has run everything sent. + void MaybeTakeOver(const GpuSplit& split, const Tegra::NativeQmdScheduler& back, + Nx2AcquireStats::QmdTakeOver where) { + if (!Current() && split.Ran(split.Position())) [[unlikely]] { + TakeOver(back, where); + } + } + void MaybeTakeOver(const GpuSplit& split, Nx2AcquireStats::QmdTakeOver where) { + MaybeTakeOver(split, back_table, where); + } + + /// TEST (EDEN_NX2_DRAIN_CHECK): the releases noted since the last call, the descriptors read + /// for them, and whether every signal since was valued; every 512th time while the view is + /// current, and after it valued signals late, a copy of its table; the position of the entry + /// whose signals wait, from the `first_waiting`-th on; the signals valued late since, with + /// what they read; nothing if no signal was sent since. + struct Checks { + std::vector releases; + std::vector reads; + std::shared_ptr table; + bool valued; + bool table_after_late; + u64 wait_position; + u32 first_waiting; + std::vector late; + }; + [[nodiscard]] std::optional TakeChecks() { + if (!checks_pending) { + return std::nullopt; + } + checks_pending = false; + Checks taken{std::move(checks), + std::move(check_reads), + nullptr, + !checks_unvalued, + false, + check_wait_position, + check_first_waiting, + std::move(late_checks)}; + if (taken.valued && Current() && (table_check_due || ++table_checks % 512 == 0)) { + auto copy = + std::make_shared(Tegra::NativeQmdScheduler::OwnTable{}); + copy->CopyStates(scheduler); + taken.table = std::move(copy); + taken.table_after_late = table_check_due; + table_check_due = false; + } + checks.clear(); + check_reads.clear(); + late_checks.clear(); + checks_unvalued = false; + check_wait_position = 0; + check_first_waiting = 0; + return taken; + } + + /// The engines schedule QMDs the front does not see (the split stopped). + void Lose() { + synced = false; + inline_writes_count = 0; + ClearWaiting(); + } + + [[nodiscard]] bool Synced() const { + return synced; + } + /// Whether the table is the back's as of everything sent: synced, and no signal waits. + [[nodiscard]] bool Current() const { + return synced && waiting.empty(); + } + /// TEST: whether the view's table holds the QMD states `other` does. + [[nodiscard]] bool SameTable(const Tegra::NativeQmdScheduler& other) const { + return scheduler.SameStates(other); + } + + /// LOAD_INLINE_QMD_DATA was just sent: running it, the back writes `qmd` to the QMD at + /// `device`. Until then the front reads it from here (ReadDescriptor). + void NoteInlineWrite(const GpuSplit& split, u64 device, const Descriptor& qmd) { + InlineWrite& written = inline_writes[inline_writes_count % inline_writes.size()]; + ++inline_writes_count; + written.device = device; + written.position = split.Position(); + written.qmd = qmd; + written.overwritten = false; + } + + /// The scheduling of PCAS action `argument` on the QMD at `address` that the back runs at the + /// entry just sent (KeplerCompute::ProcessNativeMethod); `inline_qmd`: the QMD it writes there + /// right before. Notes what the launches it runs release, or has the signal wait. The entry + /// must still be unpublished (GpuSplit::PublishHold): the back must not have run it. + template + void Signal(GpuSplit& split, const Memory& memory, GPUVAddr address, u32 argument, + const Descriptor* inline_qmd) { + split.NoteCacheWrite(); // The dispatches' shader stores + using Lost = Nx2AcquireStats::QmdLost; + checks_pending = Nx2DrainCheck::g_enabled; // TEST + const u64 position = split.Position(); + if (checks_pending) { + // TEST: the back must not have run the entry yet, the n-th signal of its entry. + if (split.Unpublished() == 0) [[unlikely]] { + Nx2DrainCheck::NotePublishedEarly(); + } + entry_signals = position != entry_position ? 0 : entry_signals + 1; + entry_position = position; + } + if (synced) { + CatchUp(split, memory); + } + if (!synced) { + Unvalued(split, Lost::NotSynced); + return; + } + if (!waiting.empty()) { + // Behind a signal still waiting. + Wait(split, position, memory.Id(), address, argument, inline_qmd, std::nullopt); + return; + } + const Outcome outcome = + Attempt(split, memory, position, address, argument, inline_qmd, false); + if (outcome.valued) { + Nx2AcquireStats::NoteQmdSignal(std::nullopt); + return; + } + if (outcome.retry_after) { + retry_after = *outcome.retry_after; + if (Nx2AcquireStats::g_enabled && outcome.lost == Lost::Pending) [[unlikely]] { + NoteWaitWriter(split, memory, position, outcome); // TEST + } + Wait(split, position, memory.Id(), address, argument, inline_qmd, outcome.lost); + return; + } + synced = false; + Unvalued(split, outcome.lost); + } + +private: + struct Release { + GPUVAddr address; + u32 payload; + u32 flags; + }; + + /// An inline QMD a LOAD_INLINE_QMD_DATA sent writes (NoteInlineWrite). + struct InlineWrite { + u64 device{}; + u64 position{}; + Descriptor qmd{}; + bool overwritten{}; ///< A release of the same entry writes into it after + }; + + /// A signal sent at `position` that waits to be valued (`inline_qmd` if `has_inline`). + struct WaitingSignal { + u64 position; + const void* memory; ///< Memory::Id of the address space it reads + GPUVAddr address; + u32 argument; + bool has_inline; + Descriptor inline_qmd; + u32 index; ///< TEST: of the signals of its entry + }; + + /// How an attempt to value a signal ended: valued, or taken back, to try again once the back + /// ran `retry_after`, or for good (the view loses its table). + struct Outcome { + bool valued; + std::optional retry_after; + Nx2AcquireStats::QmdLost lost; + u64 qmd{}; ///< TEST: the QMD it could not read + }; + + /// Signals that wait, at most: a back that far behind leaves the view to its next takeover. + static constexpr size_t MaxWaiting = 4096; + + /// A signal the front can't value, its table lost: no release of it is noted, and the view + /// takes the back's table over once the back ran it. + void Unvalued(const GpuSplit& split, Nx2AcquireStats::QmdLost lost) { + checks_unvalued = true; // TEST + Nx2AcquireStats::NoteQmdSignal(lost, WhatSent(split, lost)); + } + + /// TEST: what sent the last write the view could not read past, for `lost`. + [[nodiscard]] static UnplacedKind WhatSent(const GpuSplit& split, + Nx2AcquireStats::QmdLost lost) { + return lost == Nx2AcquireStats::QmdLost::Mappings ? split.LastMappingKind() + : split.LastUnplacedKind(); + } + + /// TEST (EDEN_NX2_ACQUIRE_STATS): what wrote the QMD a signal sent at `position` waits for, as + /// it was sent: a release into it the front noted, else the engine method whose entry ends at + /// the write's position, else another entry (a CPU function, a deferred call). Logs the first. + template + void NoteWaitWriter(const GpuSplit& split, const Memory& memory, u64 position, + const Outcome& outcome) const { + const u64 target = *outcome.retry_after; + std::string writer; + if (const auto device = memory.Device(outcome.qmd)) { + for (u64 offset = 0; offset < sizeof(Descriptor) && writer.empty(); offset += 4) { + const auto queued = split.QueuedSemaphoreWriteAt(*device + offset); + if (queued && queued->position == target) { + writer = fmt::format("a release into word {}", offset / 4); + } + } + } + if (writer.empty()) { + writer = split.MethodEndingAt(target); + } + if (writer.empty()) { + writer = "another entry"; + } + static u32 logged{}; + if (logged < 16) { + ++logged; + LOG_INFO(HW_GPU, + "QMD view: a signal waits for a write into the QMD at {:#x} by {}, sent {} " + "ring words before it; the back is {} words behind", + outcome.qmd, writer, position - target, split.Backlog()); + } + Nx2AcquireStats::NoteQmdWaitWriter(writer); + } + + /// The signal sent at `position` waits, first (`why`: what it waits for) or behind the others. + /// None of its releases is noted: they are placed once it is valued. + void Wait(const GpuSplit& split, u64 position, const void* memory_id, GPUVAddr address, + u32 argument, const Descriptor* inline_qmd, + std::optional why) { + checks_unvalued = true; // TEST + if (waiting.size() >= MaxWaiting) { + Nx2AcquireStats::NoteQmdLateLost(Nx2AcquireStats::QmdLost::TooMany, + waiting.size()); // TEST + synced = false; + ClearWaiting(); + Nx2AcquireStats::NoteQmdSignal(Nx2AcquireStats::QmdLost::NotSynced); // TEST + return; + } + Nx2AcquireStats::NoteQmdWait(why, why ? WhatSent(split, *why) : UnplacedKind::Other); + WaitingSignal& signal = waiting.emplace_back(); + signal.position = position; + signal.memory = memory_id; + signal.address = address; + signal.argument = argument; + signal.has_inline = inline_qmd != nullptr; + if (inline_qmd != nullptr) { + signal.inline_qmd = *inline_qmd; + } + if (checks_pending) { + // TEST: the back keeps what it reads for the signals of this entry from here on. + signal.index = entry_signals; + if (check_wait_position != position) { + check_wait_position = position; + check_first_waiting = entry_signals; + } + } + } + + void ClearWaiting() { + waiting.clear(); + late_reads.clear(); + retry_after = 0; + } + + /// Values the waiting signals that read `memory` in order, from the first, as long as the back + /// ran what the first waits for. + template + void CatchUp(GpuSplit& split, const Memory& memory) { + while (!waiting.empty()) { + const WaitingSignal& first = waiting.front(); + if (first.memory != memory.Id() || !split.Ran(retry_after)) { + return; + } + const Outcome outcome = + Attempt(split, memory, first.position, first.address, first.argument, + first.has_inline ? &first.inline_qmd : nullptr, true); + if (outcome.retry_after) { + retry_after = *outcome.retry_after; + return; + } + if (!outcome.valued) { + Nx2AcquireStats::NoteQmdLateLost(outcome.lost, waiting.size()); // TEST + synced = false; + ClearWaiting(); + return; + } + if (Nx2DrainCheck::g_enabled) { + // TEST: what it read, for the back to compare (CheckLateQmdReads). + late_checks.push_back({first.position, first.index, std::move(late_reads_read)}); + late_reads_read.clear(); + table_check_due = table_check_due || waiting.size() == 1; + } + waiting.pop_front(); + retry_after = 0; + Nx2AcquireStats::NoteQmdLate(waiting.empty()); // TEST + } + // Every signal sent before the reads made late is valued: none of their releases waits. + late_reads.clear(); + } + + /// Runs the signal sent at `position` on the table as the back runs it, and notes what its + /// launches release, or places it if `late`. Takes it back where the front can't read a QMD + /// as the back reads it. + template + Outcome Attempt(GpuSplit& split, const Memory& memory, u64 position, GPUVAddr address, + u32 argument, const Descriptor* inline_qmd, bool late) { + using Lost = Nx2AcquireStats::QmdLost; + // The back finds the mappings every change sent before the signal made, and none after it. + const u64 mapping_change = split.LastMappingChange(); + if (mapping_change > position) { + return {false, std::nullopt, Lost::MappedAfter}; + } + if (!split.Ran(mapping_change)) { + return {false, mapping_change, Lost::Mappings}; + } + releases.clear(); + attempt_reads.clear(); + late_reads_read.clear(); // TEST + const size_t checked_reads = check_reads.size(); // TEST + std::optional failure; + const auto read = [&](u64 qmd_address, Descriptor& descriptor) { + if (!memory.IsMapped(qmd_address, sizeof(descriptor))) { + return false; + } + if (inline_qmd != nullptr && qmd_address == address) { + descriptor = *inline_qmd; + } else if (const auto unread = + ReadDescriptor(split, memory, qmd_address, position, descriptor)) { + failure = unread; + return false; // Stops the scheduling, which is taken back + } + // A release this signal ran before writes into the QMD. + for (const Release& earlier : releases) { + if (earlier.address + 16 > qmd_address && + earlier.address < qmd_address + sizeof(descriptor)) { + failure = Outcome{false, std::nullopt, Lost::ReleasedInto}; + return false; + } + } + attempt_reads.push_back(qmd_address); + if (Nx2DrainCheck::g_enabled) { + // TEST: what the back reads too. + (late ? late_reads_read : check_reads).push_back({qmd_address, descriptor}); + } + return true; + }; + const auto release = [&](u64 destination, u32 payload, u32 flags) { + const bool four_words = (flags >> 30) == 0; + if (!memory.IsMapped(destination, four_words ? 16 : sizeof(u32))) { + return false; + } + releases.push_back({destination, payload, flags}); + return true; + }; + scheduler.BeginAttempt(); + scheduler.Signal(address, argument, read, release, [](const Descriptor&) { return true; }); + if (!failure && late && ReleasesIntoLateReads()) { + failure = Outcome{false, std::nullopt, Lost::LateReleasedInto}; + } + if (failure) { + scheduler.RollbackAttempt(); + check_reads.resize(checked_reads); // TEST: tried again later + return *failure; + } + scheduler.CommitAttempt(); + if (late) { + for (const u64 read_address : attempt_reads) { + late_reads.insert(read_address >> 8); + } + for (const Release& done : releases) { + memory.NoteWriteAt(split, done.address, (done.flags >> 30) == 0 ? 16 : 4, position); + } + } else { + for (const Release& noted : releases) { + NoteRelease(split, memory, noted.address, noted.payload, noted.flags); + } + } + return {true, std::nullopt, Lost::NotSynced}; + } + + /// Late: whether a release of the signal writes a QMD it read, or one a signal valued late + /// before read. The back may have run the release before the front read that QMD. + [[nodiscard]] bool ReleasesIntoLateReads() const { + for (const Release& done : releases) { + const u64 first = done.address >> 8; + const u64 last = (done.address + 15) >> 8; + for (const u64 read_address : attempt_reads) { + if ((read_address >> 8) == first || (read_address >> 8) == last) { + return true; + } + } + if (late_reads.contains(first) || late_reads.contains(last)) { + return true; + } + } + return false; + } + + /// Reads into `descriptor` the QMD at `qmd_address` as the back read it, or will, running the + /// signal sent at `position`, or tells why the front can't. + template + [[nodiscard]] std::optional ReadDescriptor(const GpuSplit& split, const Memory& memory, + GPUVAddr qmd_address, u64 position, + Descriptor& descriptor) const { + using Lost = Nx2AcquireStats::QmdLost; + const auto device = memory.Device(qmd_address); + if (!device || memory.GpuWritten(*device, sizeof(descriptor))) { + return Outcome{false, std::nullopt, Lost::GpuWritten}; + } + // The last write sent there, placed or not (the inline write covers the whole QMD). + const u64 target = memory.WritesTarget(split, qmd_address, sizeof(descriptor)); + const Lost lost = target == split.LastUnplacedWrite() ? Lost::Unplaced : Lost::Pending; + if (target > position) { + // Sent after the signal (it waited): memory may hold it, the back read what was before. + return Outcome{false, std::nullopt, Lost::WrittenAfter, qmd_address}; + } + if (split.Ran(target)) { + // Read only once Ran saw the back past the write: the back stores its position after + // the writes before it. A read before that look may be older than the write. The back + // can run a write published shortly before the signal (to an idle back, after a few + // words) while the front reads, and store its position before the front looks. + memory.Read(qmd_address, descriptor.data(), sizeof(descriptor)); + return std::nullopt; + } + if (const InlineWrite* written = QueuedInlineWrite(split, *device); + written != nullptr && written->position == target) { + descriptor = written->qmd; + Nx2AcquireStats::NoteQmdInlineRead(); // TEST + return std::nullopt; + } + return Outcome{false, target, lost, qmd_address}; + } + + /// The last inline QMD sent to the QMD at `device` that the back has not written yet. + [[nodiscard]] const InlineWrite* QueuedInlineWrite(const GpuSplit& split, u64 device) const { + const size_t count = (std::min)(inline_writes_count, inline_writes.size()); + for (size_t i = 1; i <= count; ++i) { + const InlineWrite& written = + inline_writes[(inline_writes_count - i) % inline_writes.size()]; + if (split.Ran(written.position)) { + break; // The back wrote it, and the older ones + } + if (written.device == device) { + return written.overwritten ? nullptr : &written; + } + } + return nullptr; + } + + /// A release the back runs with the entry just sent, as ProcessNativeMethod's: what the word + /// holds once the back ran it. + template + void NoteRelease(GpuSplit& split, const Memory& memory, GPUVAddr address, u32 payload, + u32 flags) { + const bool four_words = (flags >> 30) == 0; + const bool reduction = (flags & (1U << 26)) != 0; + const bool maximum = !reduction && !four_words && (flags & (1U << 19)) != 0; + const u64 size = four_words ? 16 : sizeof(u32); + const auto device = memory.Device(address); + // The word a reduction or a maximum combines the payload with, before this write. + std::optional base; + if (device && (reduction || maximum)) { + base = WordAt(split, memory, address, *device); + } + memory.NoteWrite(split, address, size); + if (!device) { + return; + } + // An inline QMD this entry wrote before is no longer what memory holds there. + for (size_t i = 1; i <= (std::min)(inline_writes_count, inline_writes.size()); ++i) { + InlineWrite& written = inline_writes[(inline_writes_count - i) % inline_writes.size()]; + if (written.position != split.Position()) { + break; + } + if (written.device < *device + size && *device < written.device + sizeof(Descriptor)) { + written.overwritten = true; + } + } + // TEST: what was noted (EDEN_NX2_DRAIN_CHECK, EDEN_NX2_ACQUIRE_STATS). + const auto check = [this](u64 word, u32 value, bool known, bool at_least) { + using Note = Nx2AcquireStats::QmdNote; + Nx2AcquireStats::NoteQmdNote(!known ? Note::Unknown + : at_least ? Note::AtLeast + : Note::Exact); + if (checks_pending) { + checks.push_back({word, value, known, at_least}); + } + }; + Nx2AcquireStats::NoteRelease(*device, Nx2AcquireStats::Releaser::Qmd); + if (reduction) { + const bool is_signed = ((flags >> 24) & 1) != 0; + const u32 value = + base ? Tegra::ReduceSemaphore((flags >> 20) & 7, is_signed, *base, payload) : 0; + split.NoteSemaphoreWrite(*device, value, base.has_value()); + check(*device, value, base.has_value(), false); + } else if (four_words) { + // The payload as 64 bits, a timestamp after it. Fence-flagged payload releases, like + // the plain release below (GpuSplit::FencedPayloadLanding). + const auto landing = GpuSplit::FencedPayloadLanding(); + split.NoteSemaphoreWrite(*device, payload, true, false, landing); + split.NoteSemaphoreWrite(*device + 4, 0, true, false, landing); + check(*device, payload, true, false); + check(*device + 4, 0, true, false); + } else if (maximum) { + // Whatever the word held, it holds the payload at least. + const u32 value = base ? (std::max)(*base, payload) : payload; + split.NoteSemaphoreWrite(*device, value, true, !base.has_value()); + check(*device, value, true, !base.has_value()); + } else { + split.NoteSemaphoreWrite(*device, payload, true, false, + GpuSplit::FencedPayloadLanding()); + check(*device, payload, true, false); + } + } + + /// The word the back finds at `address` (`device`) when it runs the entry just sent, where + /// the front knows it: the last release sent there, if nothing sent since may have written + /// over it, else memory, once the back ran every write sent there before the entry. The entry + /// itself is held unpublished (PublishHold): an idle back that ran it before this read would + /// leave its own increment in memory, and the front would count it twice. + template + [[nodiscard]] std::optional WordAt(const GpuSplit& split, const Memory& memory, + GPUVAddr address, u64 device) const { + // The back reads the syncpoint shim from the syncpoints. + if (memory.InSyncpointShim(address, sizeof(u32))) { + return std::nullopt; + } + // The last write sent to the word's granule, placed or not. + const u64 target = memory.WritesTarget(split, address, sizeof(u32)); + if (const auto queued = split.QueuedSemaphoreWriteAt(device)) { + // A fenced one lands once its fence ran, before or after the back reads the word. + if (queued->known && !queued->at_least && !queued->fenced && + queued->position >= target) { + return queued->payload; + } + return std::nullopt; + } + if (memory.GpuWritten(device, sizeof(u32)) || !split.Ran(target)) { + return std::nullopt; + } + return memory.ReadWord(address); + } + + Tegra::NativeQmdScheduler scheduler{Tegra::NativeQmdScheduler::OwnTable{}}; + /// The table the engines of every channel share, which the back's scheduling uses. + Tegra::NativeQmdScheduler back_table; + std::vector releases; + std::vector attempt_reads; ///< The QMDs the attempt read + bool synced{}; + /// Signals that wait to be valued, oldest first; the first may be once the back ran + /// retry_after. late_reads: the granules of the QMDs read late since the waiting began. + std::deque waiting; + u64 retry_after{}; + std::unordered_set late_reads; + /// TEST (EDEN_NX2_DRAIN_CHECK): the entry the last signal came with and its signals before + /// it; the entry whose signals wait and the first that waits there, for TakeChecks; what the + /// last late attempt read; the signals valued late since TakeChecks; a table copy is due. + u64 entry_position{}; + u32 entry_signals{}; + u64 check_wait_position{}; + u32 check_first_waiting{}; + std::vector late_reads_read; + std::vector late_checks; + bool table_check_due{}; + /// The last inline QMDs sent, oldest first from inline_writes_count on: a back a millisecond + /// behind has some ten to write, a few times that in long frames. + std::array inline_writes{}; + size_t inline_writes_count{}; + /// TEST (EDEN_NX2_DRAIN_CHECK): see TakeChecks. + std::vector checks; + std::vector check_reads; + u64 table_checks{}; + bool checks_pending{}; + bool checks_unvalued{}; +}; + +} // namespace VideoCommon diff --git a/src/video_core/gpu.cpp b/src/video_core/gpu.cpp index c044fde73f..df4a2d4883 100644 --- a/src/video_core/gpu.cpp +++ b/src/video_core/gpu.cpp @@ -35,14 +35,15 @@ #include "video_core/engines/maxwell_dma.h" #include "video_core/gpu.h" #include "video_core/pending_maps.h" +#include "video_core/gpu_split.h" #include "video_core/gpu_thread.h" #include "video_core/gpu_timestamp.h" +#include "video_core/gpu_wait_stats.h" #include "video_core/host1x/host1x.h" #include "video_core/host1x/syncpoint_manager.h" #include "video_core/memory_manager.h" #include "video_core/nx2.h" -#include "video_core/nx2_async_readback.h" -#include "video_core/nx2_profile.h" +#include "video_core/nx2_read_trace.h" #include "video_core/renderer_base.h" #include "video_core/shader_notify.h" @@ -158,7 +159,8 @@ struct GPU::Impl { template [[nodiscard]] u64 RequestSyncOperation(Func&& action) { std::unique_lock lck{sync_request_mutex}; - const u64 fence = ++last_sync_fence; + const u64 fence = last_sync_fence.load(std::memory_order_relaxed) + 1; + last_sync_fence.store(fence, std::memory_order_release); sync_requests.emplace_back(std::forward(action)); return fence; } @@ -168,19 +170,32 @@ struct GPU::Impl { return current_sync_fence.load(std::memory_order_relaxed); } + /// Returns once the request of `fence` ran, or once the GPU threads stopped for good + /// (NotifyShutdown): then it never runs. void WaitForSyncOperation(const u64 fence) { std::unique_lock lck{sync_request_mutex}; - sync_request_cv.wait(lck, [this, fence] { return CurrentSyncRequestFence() >= fence; }); + sync_request_cv.wait(lck, [this, fence] { + return CurrentSyncRequestFence() >= fence || sync_requests_dropped; + }); } void WaitForIdle() { + const u64 request = VideoCommon::GpuWaitStats::Now(); const u64 fence = RequestSyncOperation([] {}); gpu_thread.TickGPU(is_async); WaitForSyncOperation(fence); + VideoCommon::GpuWaitStats::RecordSync(VideoCommon::GpuWaitStats::Site::Idle, + VideoCommon::GpuWaitStats::Now() - request); } /// Tick pending requests within the GPU. void TickWork() { + // After every draw and dispatch, nearly always with nothing to do: every request made + // has run. One made meanwhile runs at the next tick, which its maker asks for. + if (current_sync_fence.load(std::memory_order_acquire) == + last_sync_fence.load(std::memory_order_acquire)) { + return; + } std::unique_lock lck{sync_request_mutex}; while (!sync_requests.empty()) { auto request = std::move(sync_requests.front()); @@ -213,11 +228,27 @@ struct GPU::Impl { /// core timing events. void Start() { Settings::UpdateGPUAccuracy(); + // With the GPU thread split, its back runs a sync operation a guest thread waits for as + // soon as it asks (OnCPURead). + VideoCommon::GpuSplit::Get().SetTickHook( + [](void* argument) { static_cast(argument)->TickWork(); }, this); gpu_thread.StartThread(*renderer, renderer->Context(), scheduler); } void NotifyShutdown() { gpu_thread.NotifyShutdown(); + VideoCommon::GpuSplit::Get().SetTickHook(nullptr, nullptr); + { + // The GPU threads stopped: no sync operation runs anymore. Guest threads still run + // until the kernel suspends them, and one waiting for a read's flush (OnCPURead) + // would wait forever, in host code where the kernel cannot suspend it + // (KernelCore::SuspendEmulation then waits for it forever). Drop the requests, which + // may refer to their waiters' frames, and release the waiters. + std::scoped_lock requests_lock{sync_request_mutex}; + sync_requests.clear(); + sync_requests_dropped = true; + } + sync_request_cv.notify_all(); std::unique_lock lk{sync_mutex}; shutting_down.store(true, std::memory_order::relaxed); sync_cv.notify_all(); @@ -265,7 +296,12 @@ struct GPU::Impl { } VideoCore::RasterizerDownloadArea OnCPURead(DAddr addr, u64 size) { + namespace Stats = VideoCommon::GpuWaitStats; + const size_t core = system.GetCurrentHostThreadID(); + const u64 check_start = Stats::Now(); auto raster_area = renderer->ReadRasterizer()->GetFlushArea(addr, size); + const u64 request = Stats::Now(); + Stats::AddAreaCheck(core, request - check_start); if (raster_area.preemtive) { return raster_area; } @@ -277,34 +313,63 @@ struct GPU::Impl { raster_area.start_address, raster_area.end_address, reads.load(std::memory_order_relaxed)); } - namespace AsyncReadback = VideoCore::AsyncReadback; - const u64 request = AsyncReadback::Now(); - const u64 area_size = raster_area.end_address - raster_area.start_address; + // Filled on the GPU thread while this thread waits for it. + Stats::FlushTiming timing{}; + // TEST (EDEN_NX2_READ_TRACE): this read on the host GPU's timeline. + namespace ReadTrace = VideoCommon::ReadTrace; + const bool traced = ReadTrace::Enabled(); + ReadTrace::Read trace{}; + if (traced) [[unlikely]] { + trace.area = raster_area.start_address; + trace.in_flight = raster_area.in_flight; + trace.request = request; + renderer->ReadRasterizer()->TraceRead(trace, ReadTrace::Step::Request); + } // In flight: an earlier read's downloads cover the area, only FinishRead is left. if (!raster_area.in_flight) { - const u64 fence = RequestSyncOperation([this, raster_area, area_size]() { - const u64 start = AsyncReadback::Now(); + const u64 fence = RequestSyncOperation([this, raster_area, &timing]() { + timing.start = Stats::Now(); + const Stats::HostWaits host_before = Stats::t_host_waits; + const u64 bytes_before = Stats::t_download_bytes; // Apply collected guest writes before downloading GPU data over guest memory. InvalidateGPUCache(); - renderer->ReadRasterizer()->FlushRegionForRead(raster_area.start_address, - area_size); - AsyncReadback::Add(AsyncReadback::g_counters.flush_ns, - AsyncReadback::Now() - start); + timing.applied = Stats::Now(); + renderer->ReadRasterizer()->FlushRegionForRead( + raster_area.start_address, raster_area.end_address - raster_area.start_address); + timing.host_ns = Stats::t_host_waits.ns - host_before.ns; + timing.finishes = Stats::t_host_waits.count - host_before.count; + timing.bytes = Stats::t_download_bytes - bytes_before; + timing.end = Stats::Now(); }); + // The split's back runs it after the entry it is on. The tick sent through the GPU + // thread's queue still runs it without the split. + VideoCommon::GpuSplit::Get().RequestTick(); gpu_thread.TickGPU(is_async); + if (traced) [[unlikely]] { + // While the GPU thread runs the flush. + renderer->ReadRasterizer()->TraceRead(trace, ReadTrace::Step::Calibrate); + } WaitForSyncOperation(fence); + } else if (traced) [[unlikely]] { + renderer->ReadRasterizer()->TraceRead(trace, ReadTrace::Step::Calibrate); } // The GPU thread submitted the downloads without waiting for the host GPU: this thread // waits for them and writes them to guest memory, while the GPU thread goes on. - AsyncReadback::t_reading = true; - renderer->ReadRasterizer()->FinishRead(raster_area.start_address, area_size); - AsyncReadback::t_reading = false; - AsyncReadback::CountRead(raster_area.in_flight ? AsyncReadback::Read::InFlight - : AsyncReadback::Read::Async, - AsyncReadback::Now() - request); - if (VideoCommon::Nx2Profile::Enabled()) { - AsyncReadback::MaybeReport(); + const u64 finish_start = Stats::Now(); + if (traced) [[unlikely]] { + ReadTrace::t_read = &trace; + } + renderer->ReadRasterizer()->FinishRead(raster_area.start_address, + raster_area.end_address - raster_area.start_address); + const u64 woke = Stats::Now(); + if (traced) [[unlikely]] { + ReadTrace::t_read = nullptr; + trace.landed = woke; + renderer->ReadRasterizer()->TraceRead(trace, ReadTrace::Step::Done); } + timing.finish_ns = woke - finish_start; + timing.in_flight = raster_area.in_flight; + Stats::RecordReactive(raster_area, core, request, timing, woke); return raster_area; } @@ -313,12 +378,15 @@ struct GPU::Impl { if (preserve_gpu_writes) { VideoCore::RasterizerInterface* rasterizer = renderer->ReadRasterizer(); if (!rasterizer->TryCacheInvalidation(addr, size)) { + const u64 request = VideoCommon::GpuWaitStats::Now(); const u64 fence = RequestSyncOperation([rasterizer, addr, size] { rasterizer->FlushRegion(addr, size, VideoCommon::CacheType::BufferCache); rasterizer->OnCacheInvalidation(addr, size); }); gpu_thread.TickGPU(is_async); WaitForSyncOperation(fence); + VideoCommon::GpuWaitStats::RecordSync(VideoCommon::GpuWaitStats::Site::Invalidate, + VideoCommon::GpuWaitStats::Now() - request); } return; } @@ -424,11 +492,13 @@ struct GPU::Impl { std::list> sync_requests; std::atomic current_sync_fence{}; std::atomic ampere_anchor{}; - u64 last_sync_fence{}; + std::atomic last_sync_fence{}; ///< The last request's fence (written under the mutex) VideoCommon::PendingMaps pending_maps; std::mutex sync_request_mutex; std::condition_variable sync_request_cv; + /// The GPU threads stopped and the requests were dropped (NotifyShutdown, under the mutex). + bool sync_requests_dropped{}; /// InvalidateGPUCache: the writes it gathered and is applying, on each thread doing so. std::vector>*> applying_writes; diff --git a/src/video_core/gpu_split.cpp b/src/video_core/gpu_split.cpp index 2a7e43885d..7944c11c8d 100644 --- a/src/video_core/gpu_split.cpp +++ b/src/video_core/gpu_split.cpp @@ -2,7 +2,6 @@ // SPDX-License-Identifier: GPL-3.0-or-later #include -#include #include #include @@ -17,6 +16,7 @@ #endif #include #include +#include #include "common/dkb_cpu_trace.h" #include "common/logging.h" @@ -26,6 +26,7 @@ #include "video_core/engines/engine_interface.h" #include "video_core/engines/maxwell_3d.h" #include "video_core/gpu_split.h" +#include "video_core/gpu_wait_stats.h" #include "video_core/memory_manager.h" #include "video_core/nx2.h" #include "video_core/nx2_fast_draw.h" @@ -49,6 +50,29 @@ void Pause() { [[maybe_unused]] std::atomic g_back_thread_id{}; +/// TEST: an engine's name, by the class it was bound as (MethodEndingAt). +[[nodiscard]] std::string_view EngineName(std::optional type) { + using Tegra::Engines::EngineTypes; + if (!type) { + return "an unbound engine"; + } + switch (*type) { + case EngineTypes::Nv01Timer: + return "Nv01Timer"; + case EngineTypes::KeplerCompute: + return "KeplerCompute"; + case EngineTypes::Maxwell3D: + return "Maxwell3D"; + case EngineTypes::Fermi2D: + return "Fermi2D"; + case EngineTypes::MaxwellDMA: + return "MaxwellDMA"; + case EngineTypes::KeplerMemory: + return "KeplerMemory"; + } + return "an engine"; +} + } // Anonymous namespace void GpuSplit::CheckOwner(const char* where) { @@ -117,13 +141,12 @@ void GpuSplit::MaybeStart(Core::System& system_) { } system = &system_; check_owner = std::getenv("EDEN_NX2_SPLIT_CHECK") != nullptr; - const char* const verify = std::getenv("EDEN_NX2_ACQUIRE_VERIFY"); - g_acquire_verify = verify != nullptr && verify[0] == '1'; - g_acquire_counting = g_acquire_verify || Nx2Profile::Enabled(); + profile_publishes = Nx2Profile::Enabled(); // TEST + log_methods = std::getenv("EDEN_NX2_ACQUIRE_STATS") != nullptr; // TEST ring = std::make_unique(RingWords); + ++starts; // Before the back starts: it tags the entries of fenced writes with it (EntryTag) back_thread = std::jthread([this](std::stop_token stop_token) { BackLoop(stop_token); }); active = true; - ++starts; const char* jit = std::getenv("EDEN_NX2_MME_JIT"); front_macros = mode == '2' && (jit == nullptr || jit[0] != '0'); LOG_INFO(HW_GPU, "GPU thread split: engine methods run on a second thread{}", @@ -148,12 +171,21 @@ void GpuSplit::Stop() { last_writer_position = 0; last_sync_position = 0; synced_writer_position = 0; + last_entry_start = 0; + back_entry_start = 0; semaphore_writes.clear(); semaphore_writes_limit = 1024; last_any_write = 0; + last_unplaced_write = 0; + last_mapping_change = 0; + last_unplaced_kind = UnplacedKind::Other; + last_mapping_kind = UnplacedKind::Other; last_noted_write = 0; - write_positions.fill(0); + write_sets.fill({}); + coarse_sets.fill({}); published_local = 0; + publish_held = false; + logged_method_count = 0; consumed_cache = 0; publish_every = PublishEvery; front_engine = nullptr; @@ -164,6 +196,7 @@ void GpuSplit::Stop() { back_idle_at.store(~u64{0}, std::memory_order_relaxed); back_waiting.store(false, std::memory_order_relaxed); front_waiting.store(false, std::memory_order_relaxed); + tick_requested.store(false, std::memory_order_relaxed); } u64* GpuSplit::ReserveSlow(size_t words) { @@ -201,10 +234,35 @@ void GpuSplit::WaitForSpace(size_t words) { } } +void GpuSplit::NoteFirstUnpublished() { + first_unpublished_ticks = Nx2Profile::Ticks(); +} + +void GpuSplit::NoteEarlyPublish() { + Nx2Profile::Add(Nx2Profile::Counter::EarlyPublishes, 1); +} + +void GpuSplit::NoteLatePublish() { + const u64 idle_since = back_idle_since.load(std::memory_order_relaxed); + if (idle_since == 0 || first_unpublished_ticks == 0) { + return; + } + // The back could have run the words since it ran out of work or since the first was sent. + const u64 since = std::max(idle_since, first_unpublished_ticks); + const u64 now = Nx2Profile::Ticks(); + if (now > since) { + Nx2Profile::Add(Nx2Profile::Counter::LatePublishTicks, now - since); + Nx2Profile::Add(Nx2Profile::Counter::LatePublishes, 1); + } +} + void GpuSplit::Publish() { if (published_local == write_position) { return; } + if (profile_publishes) [[unlikely]] { + NoteLatePublish(); // TEST + } published_local = write_position; publish_every = std::min(publish_every * 2, PublishEvery); published.store(write_position, std::memory_order_seq_cst); @@ -216,34 +274,18 @@ void GpuSplit::Publish() { void GpuSplit::Drain(DrainSite site) { consumed_cache = consumed.load(std::memory_order_acquire); - if (consumed_cache == write_position && BackQuiet()) { + if (consumed_cache == write_position) { return; } Publish(); Nx2Profile::Scope scope{Nx2Profile::Zone::SplitDrain}; const u64 start = Nx2Profile::Enabled() ? Nx2Profile::Ticks() : 0; - // Everything consumed is not enough: the back submits deferred work (Scheduler::DeferFlush) - // from its idle hook afterwards, and the caller may use the scheduler once this returns. - bool hook_asked = false; for (u32 spins = 0;; ++spins) { consumed_cache = consumed.load(std::memory_order_acquire); if (consumed_cache == write_position) { - if (BackQuiet()) { - publish_every = 8; - Nx2Profile::AddDrain(site, start); - return; - } - if (!hook_asked) { - RequestIdleHook(); - hook_asked = true; - } - // Backoff sleeps only while entries are unconsumed: spin until the hook is done. - if (spins < 16384) { - Pause(); - } else { - std::this_thread::yield(); - } - continue; + publish_every = PublishAfterWait; + Nx2Profile::AddDrain(site, start); + return; } Backoff(spins, write_position); } @@ -257,26 +299,57 @@ bool GpuSplit::WaitForSync() { return WaitFor(std::min(last_sync_position, write_position), DrainSite::ListSync); } -void GpuSplit::NoteSemaphoreWrite(u64 device_address, u32 payload, bool known) { +bool GpuSplit::WaitForSyncedWriters(DrainSite site) { + return WaitFor(std::min(synced_writer_position, write_position), site); +} + +GpuSplit::Landing GpuSplit::FencedPayloadLanding() { + // As QueryCacheBase::CounterReport decides is_synced. + const bool synced = Settings::IsGPUFenceBehaviorDefault() + ? !Settings::IsGPULevelHigh() + : !Settings::IsGPUFenceBehaviorBalanced() && + !Settings::IsGPUFenceBehaviorAccurate(); + return synced ? Landing::AtRun : Landing::Fenced; +} + +void GpuSplit::NoteSemaphoreWrite(u64 device_address, u32 payload, bool known, bool at_least, + Landing landing) { if (semaphore_writes.size() >= semaphore_writes_limit) { - // The words of releases the back ran hold their values in memory. - const u64 done = consumed.load(std::memory_order_acquire); + // The words of releases in memory hold their values. std::erase_if(semaphore_writes, - [done](const auto& entry) { return entry.second.position <= done; }); + [this](const auto& entry) { return InMemory(entry.second); }); semaphore_writes_limit = std::max(1024, semaphore_writes.size() * 2); } - semaphore_writes.insert_or_assign(device_address & ~u64{3}, - SemaphoreWrite{write_position, payload, known}); + // The release is in the last entry sent, or one before (a QMD signal valued late). + SemaphoreWrite write{write_position, EntryTag(last_entry_start), payload, + known, at_least, landing == Landing::Fenced}; + const u64 word = device_address & ~u64{3}; + if (const auto it = semaphore_writes.find(word); + !write.fenced && it != semaphore_writes.end() && it->second.fenced && + fenced_landed.load(std::memory_order_acquire) <= it->second.entry) { + // A fenced write sent before may land after this one: until it did, the word holds + // either. + write.entry = it->second.entry; + write.known = false; + write.at_least = false; + write.fenced = true; + } + semaphore_writes.insert_or_assign(word, write); } std::optional GpuSplit::QueuedSemaphoreWriteAt( u64 device_address) const { const auto it = semaphore_writes.find(device_address & ~u64{3}); - if (it == semaphore_writes.end() || - it->second.position <= consumed.load(std::memory_order_acquire)) { + if (it == semaphore_writes.end() || InMemory(it->second)) { return std::nullopt; } - return QueuedSemaphoreWrite{it->second.payload, it->second.known}; + return QueuedSemaphoreWrite{it->second.payload, it->second.known, it->second.at_least, + it->second.fenced, it->second.position}; +} + +bool GpuSplit::WritersPending() const { + return consumed.load(std::memory_order_acquire) < + std::min(last_writer_position, write_position); } namespace { @@ -300,20 +373,73 @@ void ForEachDeviceRange(const Tegra::MemoryManager& memory, GPUVAddr address, u6 } // Anonymous namespace void GpuSplit::NoteWrite(const Tegra::MemoryManager& memory, GPUVAddr address, u64 size) { - if (size > MaxNotedBytes || address + size < address) { - NoteAnyWrite(); + NoteWriteAt(memory, address, size, write_position); +} + +void GpuSplit::NoteWriteAt(const Tegra::MemoryManager& memory, GPUVAddr address, u64 size, + u64 position) { + if (size == 0) { + return; + } + if (size > MaxCoarseNotedBytes || address + size < address) { + NoteUnplacedWrite(); + TagUnplaced(UnplacedKind::Oversized); // TEST return; } - ForEachDeviceRange(memory, address, size, - [this](DAddr device, u64 bytes) { NoteDeviceWrite(device, bytes); }); + if (size > MaxNotedBytes) { + ForEachDeviceRange(memory, address, size, [this, position](DAddr device, u64 bytes) { + NoteCoarseDeviceWriteAt(device, bytes, position); + }); + return; + } + ForEachDeviceRange(memory, address, size, [this, position](DAddr device, u64 bytes) { + NoteDeviceWriteAt(device, bytes, position); + }); +} + +void GpuSplit::NoteSetWrite(WriteSet& set, u64 granule, u64 position) { + const u32 tag = static_cast(granule); + size_t oldest = 0; + for (size_t way = 0; way < WriteWays; ++way) { + if (set.granules[way] == tag) { + set.positions[way] = std::max(set.positions[way], position); + return; + } + if (set.positions[way] < set.positions[oldest]) { + oldest = way; + } + } + // The granule's earlier writes, if any, are among the evicted ones. + const u64 granule_before = set.evicted; + set.evicted = std::max(set.evicted, set.positions[oldest]); + set.granules[oldest] = tag; + set.positions[oldest] = std::max(position, granule_before); +} + +u64 GpuSplit::SetTarget(const WriteSet& set, u64 granule) { + const u32 tag = static_cast(granule); + for (size_t way = 0; way < WriteWays; ++way) { + if (set.granules[way] == tag) { + return set.positions[way]; + } + } + return set.evicted; } -void GpuSplit::NoteDeviceWrite(DAddr address, u64 size) { +void GpuSplit::NoteDeviceWriteAt(DAddr address, u64 size, u64 position) { const u64 last = (address + size - 1) >> WriteGranuleBits; for (u64 granule = address >> WriteGranuleBits; granule <= last; ++granule) { - write_positions[WriteSlot(granule)] = write_position; + NoteSetWrite(write_sets[WriteSlot(granule)], granule, position); + } + last_noted_write = std::max(last_noted_write, position); +} + +void GpuSplit::NoteCoarseDeviceWriteAt(DAddr address, u64 size, u64 position) { + const u64 last = (address + size - 1) >> CoarseGranuleBits; + for (u64 granule = address >> CoarseGranuleBits; granule <= last; ++granule) { + NoteSetWrite(coarse_sets[CoarseSlot(granule)], granule, position); } - last_noted_write = write_position; + last_noted_write = std::max(last_noted_write, position); } u64 GpuSplit::WritesTarget(const Tegra::MemoryManager& memory, GPUVAddr address, u64 size) const { @@ -327,11 +453,28 @@ u64 GpuSplit::WritesTarget(const Tegra::MemoryManager& memory, GPUVAddr address, return target; } +u64 GpuSplit::MemoryWritesTarget(const Tegra::MemoryManager& memory, GPUVAddr address, + u64 size) const { + u64 target = last_unplaced_write; + if (size > MaxNotedBytes || address + size < address) { + target = std::max(target, last_noted_write); + } else { + ForEachDeviceRange(memory, address, size, [this, &target](DAddr device, u64 bytes) { + target = std::max(target, DeviceWritesTarget(device, bytes)); + }); + } + return std::min(target, write_position); +} + u64 GpuSplit::DeviceWritesTarget(DAddr address, u64 size) const { u64 target = 0; const u64 last = (address + size - 1) >> WriteGranuleBits; for (u64 granule = address >> WriteGranuleBits; granule <= last; ++granule) { - target = std::max(target, write_positions[WriteSlot(granule)]); + target = std::max(target, SetTarget(write_sets[WriteSlot(granule)], granule)); + } + const u64 last_coarse = (address + size - 1) >> CoarseGranuleBits; + for (u64 granule = address >> CoarseGranuleBits; granule <= last_coarse; ++granule) { + target = std::max(target, SetTarget(coarse_sets[CoarseSlot(granule)], granule)); } return target; } @@ -341,10 +484,6 @@ bool GpuSplit::WaitForWrites(const Tegra::MemoryManager& memory, GPUVAddr addres DrainSite::MacroDmaWrites); } -bool GpuSplit::WaitForSyncedWriters() { - return WaitFor(std::min(synced_writer_position, write_position), DrainSite::ListWriters); -} - bool GpuSplit::WaitFor(u64 target, DrainSite site) { consumed_cache = consumed.load(std::memory_order_acquire); if (consumed_cache >= target) { @@ -358,6 +497,8 @@ bool GpuSplit::WaitFor(u64 target, DrainSite site) { consumed_cache = consumed.load(std::memory_order_acquire); if (consumed_cache >= target) { wait_target.store(~u64{0}, std::memory_order_relaxed); + // The back may run out of what was sent before long. + publish_every = PublishAfterWait; Nx2Profile::AddDrain(site, start); return true; } @@ -416,17 +557,54 @@ void GpuSplit::MultiMethod(Tegra::Engines::EngineInterface* engine, u32 method, amount -= piece; methods_pending -= piece; } - if (writes == Writes::Anywhere) { - NoteAnyWrite(); + if (log_methods) [[unlikely]] { + LogMethod(engine, method); // TEST + } + NoteWrites(writes); +} + +void GpuSplit::LogMethod(const Tegra::Engines::EngineInterface* engine, u32 method) { + if (!logged_methods) { + logged_methods = std::make_unique(LoggedMethods); + } + logged_methods[logged_method_count++ & (LoggedMethods - 1)] = {write_position, + engine->bound_type, method}; +} + +std::string GpuSplit::MethodEndingAt(u64 end) const { + if (!logged_methods || logged_method_count == 0) { + return {}; + } + // Ends grow with the count: a binary search over the ones kept. + u64 low = logged_method_count > LoggedMethods ? logged_method_count - LoggedMethods : 0; + u64 high = logged_method_count; + while (low < high) { + const u64 middle = low + (high - low) / 2; + if (logged_methods[middle & (LoggedMethods - 1)].end < end) { + low = middle + 1; + } else { + high = middle; + } + } + if (low == logged_method_count) { + return {}; + } + const SentMethod& sent = logged_methods[low & (LoggedMethods - 1)]; + if (sent.end != end) { + return {}; } + // Not typeid: without RTTI (MSVC's /GR-) it reads type data the engines don't have. + return fmt::format("{} {:#x}", EngineName(sent.engine), sent.method); } -void GpuSplit::DeferFunction(std::function&& func) { +void GpuSplit::DeferFunction(std::function&& func, Writes writes) { auto* const heap = new std::function(std::move(func)); - Defer([heap] { - (*heap)(); - delete heap; - }); + Defer( + [heap] { + (*heap)(); + delete heap; + }, + writes); } size_t GpuSplit::Execute(const u64* entry, size_t words) { @@ -555,37 +733,34 @@ void GpuSplit::BackLoop(std::stop_token stop_token) { u64 read_position = 0; while (!stop_token.stop_requested()) { Nx2Profile::Report("GPU back thread", true); + MaybeTick(); u64 available = published.load(std::memory_order_acquire); if (available == read_position) { - Nx2Profile::Scope scope{Nx2Profile::Zone::SplitWait}; - // The front publishes what it holds after a few more words from now on. + // The front publishes what it holds at its next commit (a few words) from now on. back_idle_at.store(read_position, std::memory_order_relaxed); - const auto spin_start = g_acquire_counting ? std::chrono::steady_clock::now() - : std::chrono::steady_clock::time_point{}; + Nx2Profile::Scope scope{Nx2Profile::Zone::SplitWait}; + const GpuWaitStats::IdleScope idle{GpuWaitStats::g_back_idle_ns}; + // TEST (profile_publishes): out of work from now until this block ends. + struct IdleMark { + std::atomic& since; + ~IdleMark() { + since.store(0, std::memory_order_relaxed); + } + } const idle_mark{back_idle_since}; + if (profile_publishes) { + back_idle_since.store(Nx2Profile::Ticks(), std::memory_order_relaxed); + } // The front usually sends more within microseconds: spin briefly before sleeping. for (u32 spins = 0; spins < 8192 && available == read_position && - !hook_requested.load(std::memory_order_relaxed); + !hook_requested.load(std::memory_order_relaxed) && + !tick_requested.load(std::memory_order_relaxed); ++spins) { Pause(); available = published.load(std::memory_order_acquire); } - if (g_acquire_counting) { - Count(g_acquire_stats.back_waits); - Count(g_acquire_stats.back_spin_ns, - static_cast(std::chrono::duration_cast( - std::chrono::steady_clock::now() - spin_start) - .count())); - } - if (available == read_position) { + if (available == read_position && idle_hook != nullptr) { hook_requested.store(false, std::memory_order_seq_cst); - // The front may already see everything consumed: Drain() waits for back_quiet - // too, so the deferred submit can't overlap whatever the front does next. - if (!back_quiet.load(std::memory_order_relaxed)) { - if (idle_hook != nullptr) { - idle_hook(idle_argument); - } - back_quiet.store(true, std::memory_order_release); - } + idle_hook(idle_argument); available = published.load(std::memory_order_acquire); } if (available == read_position) { @@ -594,10 +769,11 @@ void GpuSplit::BackLoop(std::stop_token stop_token) { back_waiting.store(true, std::memory_order_seq_cst); const u32 seen = wake.load(std::memory_order_seq_cst); available = published.load(std::memory_order_seq_cst); - // A request made after the hook ran (RequestIdleHook) runs it before sleeping. + // A request made after the hook ran (RequestIdleHook) runs it before sleeping, and + // a requested tick runs at the top of the loop. if (available == read_position && !stop_token.stop_requested() && - !hook_requested.load(std::memory_order_seq_cst)) { - Count(g_acquire_stats.back_sleeps); + !hook_requested.load(std::memory_order_seq_cst) && + !tick_requested.load(std::memory_order_seq_cst)) { wake.wait(seen, std::memory_order_seq_cst); } back_waiting.store(false, std::memory_order_relaxed); @@ -605,10 +781,9 @@ void GpuSplit::BackLoop(std::stop_token stop_token) { } } u64 last_store = read_position; - // Before any entry runs, so a front that sees these entries consumed also sees this. - back_quiet.store(false, std::memory_order_seq_cst); while (read_position != available) { const size_t index = read_position & RingMask; + back_entry_start = read_position; const size_t size = Execute(&ring[index], std::min(available - read_position, RingWords - index)); read_position += size != 0 ? size : RingWords - index; @@ -617,6 +792,10 @@ void GpuSplit::BackLoop(std::stop_token stop_token) { hook_requested.store(false, std::memory_order_seq_cst); idle_hook(idle_argument); } + // A guest thread waits for a flush: run it now, not after the draws and compute work + // still queued here, or after everything the front holds (the guest synchronized with + // the GPU work it reads before it read, so that work ran on this thread already). + MaybeTick(); // Hand ring space back now and then while working through a long batch, and at once // when the front waits for this position (WaitForWriters). if ((read_position - last_store >= 4096 || @@ -635,58 +814,4 @@ void GpuSplit::BackLoop(std::stop_token stop_token) { front_wake.notify_all(); } -void GpuSplit::ReportAcquire() { - static u32 calls{}; - static auto next = std::chrono::steady_clock::now() + std::chrono::seconds{10}; - if (++calls % 256 != 0) { - return; - } - const auto now = std::chrono::steady_clock::now(); - if (now < next) { - return; - } - next = now + std::chrono::seconds{10}; - auto& stats = g_acquire_stats; - const auto take = [](std::atomic& counter) { - return counter.exchange(0, std::memory_order_relaxed); - }; - const u64 passed = take(stats.passed); - const u64 by_queued = take(stats.by_queued); - const u64 latched = take(stats.latched); - const u64 drained = take(stats.drained); - const u64 drain_operation = take(stats.drain_operation); - const u64 drain_settings = take(stats.drain_settings); - const u64 drain_word = take(stats.drain_word); - const u64 drain_queued = take(stats.drain_queued); - const u64 valve_drains = take(stats.valve_drains); - const u64 polls = take(stats.polls); - const u64 acquire_polls = take(stats.acquire_polls); - const u64 list_polls = take(stats.list_polls); - const u64 early_publishes = take(stats.early_publishes); - const u64 settles = take(stats.settles); - const u64 back_waits = take(stats.back_waits); - const u64 back_sleeps = take(stats.back_sleeps); - const double spin_ms = static_cast(take(stats.back_spin_ns)) / 1e6; - LOG_INFO(HW_GPU, - "Acquires: {} passed without a drain ({} by a queued release), {} resumed by a " - "latch, {} drained ({} by the 1 ms valve; {} equal or mask, {} by the settings, {} " - "unmapped or GPU-written word, {} queued release unknown or short), {} polls waited " - "without a drain in 10 s; polls: {} of acquires, {} of GPU-written list copies; " - "back: {} waits for work, {} slept, {:.1f} ms spinning; {} early publishes, {} idle " - "hooks asked by polls", - passed, by_queued, latched, drained, valve_drains, drain_operation, drain_settings, - drain_word, drain_queued, polls, acquire_polls, list_polls, back_waits, back_sleeps, - spin_ms, early_publishes, settles); - const u64 mismatches = take(stats.mismatches); - const u64 unlanded = take(stats.unlanded); - const u64 late = take(stats.late); - if (mismatches != 0 || unlanded != 0 || late != 0) { - LOG_WARNING(HW_GPU, - "Acquire verify: {} passed but not met after a drain, {} met by a " - "queued release not in memory after it, {} polls waited though a drain meets " - "them, in 10 s", - mismatches, unlanded, late); - } -} - } // namespace VideoCommon diff --git a/src/video_core/gpu_split.h b/src/video_core/gpu_split.h index 5b38aa83b7..72f7f5a22a 100644 --- a/src/video_core/gpu_split.h +++ b/src/video_core/gpu_split.h @@ -6,11 +6,13 @@ #include #include #include +#include #include #include #include #include #include +#include #include #include #include @@ -27,10 +29,27 @@ class MemoryManager; namespace Tegra::Engines { class EngineInterface; +enum class EngineTypes : u32; } namespace VideoCommon { +/// TEST (EDEN_NX2_ACQUIRE_STATS): what sent the last write the front could not place +/// (GpuSplit::TagUnplaced), for why the QMD view could not value a signal. +enum class UnplacedKind : u8 { + Other, ///< A deferred call nobody tagged + RunCommand, ///< A function the CPU queued in order (unmaps, events) + Engine, ///< An engine method the front sends as it is (blits) + Oversized, ///< A placed write longer than the coarse slots take + QmdLaunch, ///< A QMD launch the view could not value + InlineQmd, ///< An inline QMD behind a change of the mappings + Count, +}; + +inline constexpr std::array(UnplacedKind::Count)> + unplaced_kind_names{"other", "CPU functions", "engines", "oversized writes", + "QMD launches", "inline QMDs"}; + /// Why the split's front waited for its back (EDEN_NX2_GPU_PROFILE counts each). enum class DrainSite : u32 { Other, @@ -48,54 +67,18 @@ enum class DrainSite : u32 { Map, ///< GPU memory maps before later command reads FrontStart, ///< The front takes an engine's macros over FrontVerify, ///< EDEN_NX2_FRONT_VERIFY - RingFull, ///< The ring is full: the front waits for the back to make room BackRegister, ///< A register a macro the back ran may have written (Maxwell3D) BackCode, ///< The macro code changes under the macros the back has yet to run + ListCopies, ///< Lists on GPU-written pages: the dispatches and copies before the last sync + RingFull, ///< The front waits for room in the split ring Count, }; -/// How semaphore acquires and the polls of suspended ones were decided, counted with -/// EDEN_NX2_GPU_PROFILE or EDEN_NX2_ACQUIRE_VERIFY and logged every 10 s (GpuSplit::ReportAcquire). -struct AcquireStats { - std::atomic passed{}; ///< Acquires and polls met without a drain - std::atomic by_queued{}; ///< Of passed: met by a release the back had not run yet - std::atomic latched{}; ///< Polls resumed by a release the back ran (no drain) - std::atomic drained{}; ///< Acquires and polls that drained the split - std::atomic drain_operation{}; ///< Of drained: an equal or mask acquire - std::atomic drain_settings{}; ///< ...the split or the accuracy settings ask for it - std::atomic drain_word{}; ///< ...the word is unmapped or on a GPU-written page - std::atomic drain_queued{}; ///< ...a queued release to it is unknown or doesn't meet it - std::atomic valve_drains{}; ///< Of drained: polls of an unmet acquire, once per 1 ms - std::atomic polls{}; ///< Polls that keep waiting without a drain - std::atomic acquire_polls{}; ///< Polls of suspended channels waiting on an acquire - std::atomic list_polls{}; ///< ...and on a command list copy the GPU writes - std::atomic early_publishes{}; ///< Publishes to a back that waits for work (Commit) - std::atomic settles{}; ///< Idle hooks asked of a back that ran everything (SettleIdleBack) - std::atomic back_waits{}; ///< The back ran out of work - std::atomic back_sleeps{}; ///< Of those: it went to sleep - std::atomic back_spin_ns{}; ///< How long it spun for more work before that - std::atomic mismatches{}; ///< VERIFY: passed, yet the word after a drain doesn't meet it - std::atomic unlanded{}; ///< VERIFY: met by a queued release not in memory after a drain - std::atomic late{}; ///< VERIFY: a poll kept waiting, yet a drain meets it -}; -inline AcquireStats g_acquire_stats; -/// Whether g_acquire_stats counts (EDEN_NX2_GPU_PROFILE or EDEN_NX2_ACQUIRE_VERIFY), as of the -/// split's last start. -inline bool g_acquire_counting{}; -/// EDEN_NX2_ACQUIRE_VERIFY=1: acquires decided without a drain drain anyway and compare. -inline bool g_acquire_verify{}; - -inline void Count(std::atomic& counter, u64 amount = 1) { - if (g_acquire_counting) { - counter.fetch_add(amount, std::memory_order_relaxed); - } -} - inline constexpr std::array(DrainSite::Count)> drain_site_names{ "other", "fetch drain", "acquire", "semaphore flush", "fence release", "list writers", "list sync", "list flush", "macro dma read", "macro dma writes", "macro dma flush", "gp sync fence", "map", "front start", "front verify", - "ring full", "back register", "back code", + "back register", "back code", "list copy writers", "ring full", }; // Splits the GPU thread in two (NX2_GPU_THREAD_SPLIT.md). The GPU thread (the front) takes the @@ -130,15 +113,29 @@ class GpuSplit { [[nodiscard]] static bool OnBackThread() { return on_back_thread; } - /// Marks the calling thread as the GPU thread (the front, or the only one without the split). static void MarkGpuThread() { on_gpu_thread = true; } - /// Whether the calling thread owns the Vulkan scheduler: the back while it runs, else the - /// GPU thread. Other threads (CPU cores, services) must not record or submit work. [[nodiscard]] static bool OwnsScheduler() { return Get().BackRunning() ? on_back_thread : on_gpu_thread; } + /// Back thread: the entry it runs, as NoteFencedWrite takes it; elsewhere nothing. + [[nodiscard]] std::optional BackEntry() const { + if (!on_back_thread) { + return std::nullopt; + } + return EntryTag(back_entry_start); + } + /// Any thread: the operation of a report the back sent in `entry` (BackEntry) ran with its + /// fence, so the report is in memory, and those of the entries before it: fences run their + /// operations in order (QueryCacheBase::CounterReport). + void NoteFencedWrite(u64 entry) { + u64 landed = fenced_landed.load(std::memory_order_relaxed); + while (landed <= entry && + !fenced_landed.compare_exchange_weak(landed, entry + 1, std::memory_order_release, + std::memory_order_relaxed)) { + } + } /// Called on the back before it sleeps waiting for work (the rasterizer submits deferred work). void SetIdleHook(void (*hook)(void*), void* argument) { idle_hook = hook; @@ -154,6 +151,25 @@ class GpuSplit { wake.fetch_add(1, std::memory_order_seq_cst); wake.notify_one(); } + /// Run on the back when a thread asks for it (RequestTick): the GPU's sync operations. + void SetTickHook(void (*hook)(void*), void* argument) { + tick_hook = hook; + tick_argument = argument; + } + /// Any thread, after queueing a sync operation it waits for: has the back run the tick hook + /// after the entry it is running, or at once if it waits for work. Otherwise the operation runs + /// after the back's next draw, or after the front sent everything it had queued before it. + void RequestTick() { + if (!back_running.load(std::memory_order_acquire)) { + return; + } + // The back checks tick_requested after setting back_waiting: one of the two sees the other. + tick_requested.store(true, std::memory_order_seq_cst); + if (back_waiting.load(std::memory_order_seq_cst)) { + wake.fetch_add(1, std::memory_order_seq_cst); + wake.notify_one(); + } + } /// Starts the back thread once the split is wanted (EDEN_NX2_SPLIT=1, a Switch 2 title, /// commands read without cache flushes). Called by the front between commands. @@ -161,10 +177,18 @@ class GpuSplit { void Stop(); /// What the sender of an entry tells about the guest memory the back writes running it, for - /// the front's reads of guest memory (WaitForWrites). + /// the front's reads of guest memory (WaitForWrites, MemoryWritesTarget). enum class Writes : u8 { - Anywhere, ///< Unknown: a read anywhere waits for the entry - Noted, ///< Nothing, or what the sender notes right after sending it (NoteWrite) + /// Unknown: a read anywhere waits for the entry, and it may change the GPU's mappings + /// (unmaps; GPU maps drain the split) + Anywhere, + /// Guest memory anywhere, the mappings not: engine work whose writes nobody placed + Unplaced, + /// The GPU caches anywhere (shader stores, copies into buffers), guest memory only through + /// them, where GPU work wrote (Nx2MayBeGpuWritten): a read of the caches waits for it, a + /// read of other guest memory not (MemoryWritesTarget) + Caches, + Noted, ///< Nothing, or what the sender notes right after sending it (NoteWrite) }; /// An engine method write, executed as DmaPusher would (method sink, ConsumeSink, call). @@ -181,9 +205,10 @@ class GpuSplit { entry[1] = *segment; } Commit(words); - if (writes == Writes::Anywhere) { - NoteAnyWrite(); + if (log_methods) [[unlikely]] { + LogMethod(engine, method); // TEST } + NoteWrites(writes); } void MultiMethod(Tegra::Engines::EngineInterface* engine, u32 method, const u32* words, u32 amount, u32 methods_pending, const GPUVAddr* segment, @@ -196,9 +221,10 @@ class GpuSplit { entry[0] = static_cast(Kind::Method) | IsLast | Direct | (u64{method & 0xffff} << 8) | (u64{argument} << 32); Commit(1); - if (writes == Writes::Anywhere) { - NoteAnyWrite(); + if (log_methods) [[unlikely]] { + LogMethod(engine, method); // TEST } + NoteWrites(writes); } /// Defer flag: the call doesn't end fast draw reuse (nx2_fast_draw.h). @@ -207,7 +233,7 @@ class GpuSplit { /// Runs `func` on the back after everything sent before it. Must be trivially copyable. /// `flags`: FastDrawNeutral when `func` can't change what a draw resolves to. template - void Defer(const Func& func, u64 flags = 0, Writes writes = Writes::Anywhere) { + void Defer(const Func& func, Writes writes = Writes::Anywhere, u64 flags = 0) { static_assert(std::is_trivially_copyable_v && alignof(Func) <= alignof(u64)); constexpr size_t payload = (sizeof(Func) + sizeof(u64) - 1) / sizeof(u64); u64* const entry = Reserve(2 + payload); @@ -215,9 +241,7 @@ class GpuSplit { entry[1] = reinterpret_cast(&CallThunk); std::memcpy(entry + 2, &func, sizeof(Func)); Commit(2 + payload); - if (writes == Writes::Anywhere) { - NoteAnyWrite(); - } + NoteWrites(writes); } /// Defer, calling `func` with a copy of `words` (a std::span). template @@ -236,11 +260,9 @@ class GpuSplit { std::memcpy(entry + 3 + function_words, words.data(), words.size_bytes()); } Commit(2 + payload); - if (writes == Writes::Anywhere) { - NoteAnyWrite(); - } + NoteWrites(writes); } - void DeferFunction(std::function&& func); + void DeferFunction(std::function&& func, Writes writes = Writes::Anywhere); /// Runs `func` on the back and waits for it: for work on back-owned state, such as a buffer /// cache download that records into the Vulkan scheduler. A drained back still submits from /// its idle hook, so the front never touches that state itself. @@ -261,30 +283,38 @@ class GpuSplit { /// Makes what was sent visible to the back. void Publish(); - /// The front has nothing to send while it polls suspended channels. A back that ran everything - /// runs its idle hook (the deferred submit) and sleeps now instead of spinning for more, as it - /// did when every poll drained (Drain asks for the hook), but without the wait. - void SettleIdleBack() { - if (hook_requested.load(std::memory_order_relaxed) || BackQuiet() || - consumed.load(std::memory_order_acquire) != write_position) { - return; - } - Count(g_acquire_stats.settles); - RequestIdleHook(); + /// The front waited for the back, which may run out of work soon: publish the next words + /// after a few, as after a drain, not after PublishEvery. + void PublishSoon() { + publish_every = PublishAfterWait; } - /// Logs g_acquire_stats every 10 s (front, while they count). - void ReportAcquire(); + /// Keeps the entries the front sends unpublished while it notes what they write from guest + /// memory as the back will find it running them (the QMD view: descriptors, and the words + /// maxima and reductions combine with). An idle back gets a published entry at once and may + /// run it before the front reads, which then reads what the entry wrote. While `hold`, Commit + /// publishes nothing; Publish (drains, waits, a full ring) still does. + class PublishHold { + public: + PublishHold(GpuSplit& split_, bool hold) : split{split_}, was_held{split_.publish_held} { + if (hold) { + split.publish_held = true; + } + } + ~PublishHold() { + if (split.publish_held != was_held) { + split.publish_held = was_held; + split.PublishIfDue(); + } + } + PublishHold(const PublishHold&) = delete; + PublishHold& operator=(const PublishHold&) = delete; + + private: + GpuSplit& split; + bool was_held; + }; /// Waits until the back executed everything sent so far. void Drain(DrainSite site = DrainSite::Other); - /// The ring position after the entries sent so far, for WaitUntil. - [[nodiscard]] u64 Position() const { - return write_position; - } - /// Waits until the back executed the entries sent before `position`. Returns whether any of - /// them was still queued. - bool WaitUntil(u64 position, DrainSite site) { - return WaitFor((std::min)(position, write_position), site); - } /// Compute or copy work was just sent: what can write GPU-written command lists. void NoteWriter() { @@ -294,37 +324,147 @@ class GpuSplit { /// Returns whether any of it was still queued. bool WaitForWriters(); /// A host-level wait was just sent (a syncpoint acquire, a host WFI, a GP entry with its sync - /// bit; semaphore acquires drain, or note it when a release still queued meets them): the - /// guest's GPU work before it is complete before the commands after it are fetched. + /// bit; semaphore acquires note it when a release still queued meets them): the guest's GPU + /// work before it is complete before the commands after it are fetched. void NoteSync() { last_sync_position = write_position; synced_writer_position = last_writer_position; } + /// The position after everything sent so far, for Ran and WaitForPosition. + [[nodiscard]] u64 Position() const { + return write_position; + } + /// Whether the back executed everything sent before `position`. It may tell so a little + /// late: the back stores its position every few thousand words and when it runs out of work. + [[nodiscard]] bool Ran(u64 position) const { + return consumed.load(std::memory_order_acquire) >= position; + } + /// The position after the last compute or copy work sent (NoteWriter). + [[nodiscard]] u64 LastWriterPosition() const { + return (std::min)(last_writer_position, write_position); + } + /// Waits until the back executed everything sent before `position`, not what came after. + /// Returns whether any of it was still queued. + bool WaitForPosition(u64 position, DrainSite site) { + return WaitFor((std::min)(position, write_position), site); + } /// Waits until the back executed the work sent before the last NoteSync. Returns whether any /// of it was still queued. bool WaitForSync(); - /// A semaphore release was just sent: once the back ran it, the word at `device_address` holds - /// `payload`, or a value the front can't tell (`known` false: a reduction, a counter report). - void NoteSemaphoreWrite(u64 device_address, u32 payload, bool known); + /// Waits until the back executed the compute and copy work sent before the last NoteSync: + /// the writers a command list fetched now is ordered after. Returns whether any of it was + /// still queued. `site`: whose wait it is, for EDEN_NX2_GPU_PROFILE. + bool WaitForSyncedWriters(DrainSite site = DrainSite::ListWriters); + /// When a release the back runs reaches memory. + enum class Landing : u8 { + AtRun, ///< As the back runs it: payload reports without a fence, reductions, maxima + Fenced, ///< Once the fence signaled after it ran (QueryCacheBase::CounterReport): + ///< fence-flagged reports, counter reports + }; + /// When a fence-flagged payload release reaches memory. Below High GPU accuracy with Default + /// fences, CounterReport keeps it synchronized (is_synced): its operation runs as the back + /// runs it, without waiting for the fence. + static Landing FencedPayloadLanding(); + /// A semaphore release was just sent: once it landed (`landing`), the word at + /// `device_address` holds `payload`, or a value the front can't tell (`known` false: a + /// reduction, a counter report), or `payload` at least (`at_least`: a maximum with a value the + /// front can't tell). + void NoteSemaphoreWrite(u64 device_address, u32 payload, bool known, bool at_least = false, + Landing landing = Landing::AtRun); struct QueuedSemaphoreWrite { u32 payload; bool known; + bool at_least; ///< Known only as a lower bound (unsigned) + /// It lands once a fence ran, at no position the front can wait for; or an earlier write + /// that does may still land after it. + bool fenced; + u64 position; ///< Once the back ran this (Ran), the word holds the write, unless `fenced` }; - /// The last write sent to the semaphore word at `device_address`, if the back may not have run - /// it yet. + /// The last write sent to the semaphore word at `device_address`, if it may not be in memory + /// yet. [[nodiscard]] std::optional QueuedSemaphoreWriteAt( u64 device_address) const; - /// Waits until the back executed the compute and copy work sent before the last NoteSync: - /// the writers a command list fetched now is ordered after. Returns whether any of it was - /// still queued. - bool WaitForSyncedWriters(); + /// Whether compute or copy work sent so far is still queued (EDEN_NX2_DRAIN_CHECK). + [[nodiscard]] bool WritersPending() const; + /// TEST (EDEN_NX2_ACQUIRE_STATS): the engine and method of the method entry that ends at + /// `end`, among the last LoggedMethods sent, as "KeplerCompute 0x6d"; empty if none. + [[nodiscard]] std::string MethodEndingAt(u64 end) const; + /// Front thread: the ring words sent that the back had not run when it last stored its + /// position (every 4096 words of a batch and at its end) (EDEN_NX2_LIST_STATS). + [[nodiscard]] u64 Backlog() const { + return write_position - consumed.load(std::memory_order_acquire); + } + /// Front thread: the ring words sent that the back does not see yet (EDEN_NX2_LIST_STATS). + [[nodiscard]] u64 Unpublished() const { + return write_position - published_local; + } - /// The entry just sent (Writes::Noted) writes [address, address + size) of `memory`. + /// The entry just sent (Writes::Noted) writes [address, address + size) of `memory`, as + /// mapped now. Up to MaxNotedBytes by 256-byte granule, up to MaxCoarseNotedBytes by 64 KiB + /// granule, longer writes count as unplaced. void NoteWrite(const Tegra::MemoryManager& memory, GPUVAddr address, u64 size); - /// The entry just sent may write guest memory anywhere. + /// NoteWrite for the entry sent at `position` (the one before it ends there), placed late: the + /// front's QMD view values a launch once the back ran what it waited for. + void NoteWriteAt(const Tegra::MemoryManager& memory, GPUVAddr address, u64 size, u64 position); + /// The entry just sent may write guest memory anywhere and change the GPU's mappings + /// (Writes::Anywhere). void NoteAnyWrite() { + NoteUnplacedWrite(); + last_mapping_change = write_position; + last_mapping_kind = UnplacedKind::Other; // TEST + } + /// The entry just sent may write guest memory anywhere (Writes::Unplaced). + void NoteUnplacedWrite() { + last_any_write = write_position; + last_unplaced_write = write_position; + last_unplaced_kind = UnplacedKind::Other; // TEST + } + /// TEST: what sent the entry just sent, if it may write anywhere or change the mappings. + void TagUnplaced(UnplacedKind kind) { + if (last_unplaced_write == write_position) { + last_unplaced_kind = kind; + } + if (last_mapping_change == write_position) { + last_mapping_kind = kind; + } + } + /// TEST: what sent the last entry that may write anywhere (LastUnplacedWrite), and the last + /// that may change the mappings (LastMappingChange). + [[nodiscard]] UnplacedKind LastUnplacedKind() const { + return last_unplaced_kind; + } + [[nodiscard]] UnplacedKind LastMappingKind() const { + return last_mapping_kind; + } + /// The entry just sent may write the GPU caches anywhere (Writes::Caches). + void NoteCacheWrite() { last_any_write = write_position; } + void NoteWrites(Writes writes) { + if (writes == Writes::Anywhere) { + NoteAnyWrite(); + } else if (writes == Writes::Unplaced) { + NoteUnplacedWrite(); + } else if (writes == Writes::Caches) { + NoteCacheWrite(); + } + } + /// The position after the last entry sent that may write guest memory where the front does + /// not know (NoteAnyWrite, NoteUnplacedWrite), outside what the caches hold. + [[nodiscard]] u64 LastUnplacedWrite() const { + return (std::min)(last_unplaced_write, write_position); + } + /// The position after the last entry sent that may change the GPU's mappings (NoteAnyWrite): + /// once the back ran it, what is mapped now is what the back finds mapped. + [[nodiscard]] u64 LastMappingChange() const { + return (std::min)(last_mapping_change, write_position); + } + /// Once the back ran everything before the position returned, no entry sent so far writes + /// guest memory in [address, address + size) of `memory` but through the GPU caches + /// (Writes::Caches): what the back reads there without them, as the QMD scheduler does, is in + /// memory then. Placed as WaitForWrites places writes. + [[nodiscard]] u64 MemoryWritesTarget(const Tegra::MemoryManager& memory, GPUVAddr address, + u64 size) const; /// Waits until the back executed every entry sent so far that can write guest memory in /// [address, address + size) of `memory`, not the others. Returns whether one was queued. /// Writes are placed by the device memory the GPU addresses map to when noted and read: GPU @@ -349,20 +489,53 @@ class GpuSplit { static constexpr size_t RingMask = RingWords - 1; static constexpr u32 MaxMultiWords = 8192; static constexpr size_t PublishEvery = 256; - /// Words a back waiting for work gets without a publish_every (Commit). static constexpr size_t PublishAfterWait = 8; - /// NoteWrite places writes by 256-byte granule of device memory, up to MaxNotedBytes long - /// (longer ones count as anywhere), in write_positions slots hashed by granule: granules that - /// share a slot share its position, so a read may wait for a write elsewhere, never miss one. + /// NoteWrite places writes by 256-byte granule of device memory, up to MaxNotedBytes long, + /// in sets of WriteWays slots hashed by granule. Writes up to MaxCoarseNotedBytes (copies, + /// uploads) go by 64 KiB granule into coarse sets, longer ones count as unplaced. A slot holds + /// a granule and the position of its last write; a write to a granule its set lacks takes the + /// slot written longest ago, and the set keeps the latest position of the writes it so + /// dropped (`evicted`). A read waits for its granule's own last write while the set holds + /// it, else for the evicted ones: it may wait for a write elsewhere, never miss one, and the + /// writes to other granules hashed there (the constant buffer words of queued draws) hold up + /// a read of a QMD only once they pushed it out of its set. static constexpr u32 WriteGranuleBits = 8; static constexpr u64 MaxNotedBytes = 0x4000; - static constexpr u32 WriteSlotBits = 10; - + static constexpr u32 WriteSlotBits = 12; ///< Sets + static constexpr u32 CoarseGranuleBits = 16; + static constexpr u64 MaxCoarseNotedBytes = u64{16} << 20; + static constexpr u32 CoarseSlotBits = 10; + static constexpr size_t WriteWays = 4; + + /// One cache line. + struct WriteSet { + std::array positions; + /// The granules' low 32 bits: granules that agree in them share a slot. + std::array granules; + u64 evicted; + u64 unused; + }; + static_assert(sizeof(WriteSet) == 64); + /// The set a granule is hashed to. static size_t WriteSlot(u64 granule) { return static_cast((granule * 0x9E3779B97F4A7C15ULL) >> (64 - WriteSlotBits)); } - void NoteDeviceWrite(DAddr address, u64 size); + static size_t CoarseSlot(u64 granule) { + return static_cast((granule * 0x9E3779B97F4A7C15ULL) >> (64 - CoarseSlotBits)); + } + /// A write to `granule` at `position` (the newest, or an earlier one placed late). + static void NoteSetWrite(WriteSet& set, u64 granule, u64 position); + /// The latest position of the writes to `granule` in its set, at most. + [[nodiscard]] static u64 SetTarget(const WriteSet& set, u64 granule); + void NoteDeviceWrite(DAddr address, u64 size) { + NoteDeviceWriteAt(address, size, write_position); + } + void NoteDeviceWriteAt(DAddr address, u64 size, u64 position); + void NoteCoarseDeviceWrite(DAddr address, u64 size) { + NoteCoarseDeviceWriteAt(address, size, write_position); + } + void NoteCoarseDeviceWriteAt(DAddr address, u64 size, u64 position); /// The position WaitForWrites waits for. [[nodiscard]] u64 WritesTarget(const Tegra::MemoryManager& memory, GPUVAddr address, u64 size) const; @@ -391,21 +564,56 @@ class GpuSplit { } u64* ReserveSlow(size_t words); void Commit(size_t words) { + if (profile_publishes && write_position == published_local) [[unlikely]] { + NoteFirstUnpublished(); // TEST + } + last_entry_start = write_position; write_position += words; + if (PublishDue() && !publish_held) { + PublishNow(); + } + } + /// Whether Commit publishes now. + [[nodiscard]] bool PublishDue() const { const u64 unpublished = write_position - published_local; - if (unpublished >= publish_every) { - Publish(); - return; + // The back waits for work at the last position published (it stored it in back_idle_at + // before it spun): it gets these words after a few, not after publish_every. + return unpublished >= publish_every || + (unpublished >= PublishAfterWait && + back_idle_at.load(std::memory_order_relaxed) == published_local); + } + void PublishNow() { + if (profile_publishes && write_position - published_local < publish_every) [[unlikely]] { + NoteEarlyPublish(); // TEST } - // The back waits for work at the last position published (back_idle_at): it gets these - // words after a few, not after publish_every. A drain used to restart it like that. - if (unpublished >= PublishAfterWait && - back_idle_at.load(std::memory_order_relaxed) == published_local) { - Count(g_acquire_stats.early_publishes); - Publish(); + Publish(); + } + /// What Commit would have published during a PublishHold. + void PublishIfDue() { + if (PublishDue()) { + PublishNow(); } } + /// TEST (EDEN_NX2_GPU_PROFILE): how long the back waited for work while the front held words + /// it could have run (Publish). + /// Nx2Profile::Enabled() as the split started (MaybeStart, after VideoCore::CreateGPU read + /// the switch). + bool profile_publishes{}; + void NoteFirstUnpublished(); + void NoteLatePublish(); + /// TEST: Commit publishes before publish_every, because the back waits for the words. + void NoteEarlyPublish(); void WaitForSpace(size_t words); + /// TEST (EDEN_NX2_ACQUIRE_STATS): the engine method entries sent, by the position after them. + struct SentMethod { + u64 end; + std::optional engine; ///< The class it was bound as + u32 method; + }; + static constexpr size_t LoggedMethods = size_t{1} << 16; + /// EDEN_NX2_ACQUIRE_STATS as the split started (MaybeStart). + bool log_methods{}; + void LogMethod(const Tegra::Engines::EngineInterface* engine, u32 method); void SelectEngine(Tegra::Engines::EngineInterface* engine) { if (engine != front_engine) [[unlikely]] { SwitchEngine(engine); @@ -427,6 +635,14 @@ class GpuSplit { void BackLoop(std::stop_token stop_token); /// Runs the entry at `entry` (of the `words` published before the ring's end), returns its size. size_t Execute(const u64* entry, size_t words); + /// The back, between entries: runs the tick hook if a thread asked for it. + void MaybeTick() { + if (tick_requested.load(std::memory_order_relaxed) && + tick_requested.exchange(false, std::memory_order_acq_rel) && tick_hook != nullptr) + [[unlikely]] { + tick_hook(tick_argument); + } + } Core::System* system{}; bool active{}; @@ -439,33 +655,59 @@ class GpuSplit { u64 last_writer_position{}; u64 last_sync_position{}; u64 synced_writer_position{}; + u64 last_entry_start{}; ///< Where the last entry sent starts struct SemaphoreWrite { u64 position; ///< write_position once it was sent + /// `fenced`: the entry (EntryTag) whose fenced write must have landed (NoteFencedWrite) + u64 entry; u32 payload; bool known; + bool at_least; + bool fenced; }; /// The front's NoteSemaphoreWrite, by word address. std::unordered_map semaphore_writes; size_t semaphore_writes_limit = 1024; ///< Forget what the back ran past this many - /// After the last entry that may write guest memory anywhere (NoteAnyWrite), the last one - /// NoteWrite placed, and the last placed write in each slot's granules. + /// After the last entry that may write guest memory or the GPU caches anywhere (NoteAnyWrite, + /// NoteUnplacedWrite, NoteCacheWrite), the last one that may write guest memory itself + /// anywhere (NoteAnyWrite, NoteUnplacedWrite), the last one that may change the GPU's + /// mappings (NoteAnyWrite), the last one NoteWrite placed, and the last placed write in each + /// slot's granules, fine and coarse. u64 last_any_write{}; + u64 last_unplaced_write{}; + u64 last_mapping_change{}; + UnplacedKind last_unplaced_kind{}; ///< TEST: see TagUnplaced + UnplacedKind last_mapping_kind{}; ///< TEST u64 last_noted_write{}; - std::array write_positions{}; + alignas(64) std::array write_sets{}; + alignas(64) std::array coarse_sets{}; u64 published_local{}; + /// PublishHold: Commit publishes nothing. + bool publish_held{}; u64 consumed_cache{}; - /// Words between publishes: small right after a drain, while the back has nothing to do. + u64 first_unpublished_ticks{}; ///< TEST: when the first word not published yet was sent + std::unique_ptr logged_methods; ///< TEST: see LogMethod + u64 logged_method_count{}; + /// Words between publishes: small right after the front waited for the back (Drain, + /// PublishSoon), while the back may have nothing to do. size_t publish_every{PublishEvery}; Tegra::Engines::EngineInterface* front_engine{}; // Back side. Tegra::Engines::EngineInterface* back_engine{}; + u64 back_entry_start{}; ///< Where the entry the back runs starts (BackEntry) - /// Whether the back is done with the Vulkan scheduler until it gets more work: set after its - /// idle hook ran, cleared before it executes entries. - [[nodiscard]] bool BackQuiet() const { - return back_quiet.load(std::memory_order_acquire) || - !back_running.load(std::memory_order_acquire); + /// An entry's position as fenced writes name it: with this start of the split in the high + /// bits, so a fence of an earlier start that runs late lands nothing of this one (positions + /// stay far below 2^48). + [[nodiscard]] u64 EntryTag(u64 position) const { + return (u64{starts} << 48) | position; + } + /// Whether the word holds `write` (as far as no other write changed it since): the back ran + /// it, and a fenced one landed. + [[nodiscard]] bool InMemory(const SemaphoreWrite& write) const { + return write.position <= consumed.load(std::memory_order_acquire) && + (!write.fenced || fenced_landed.load(std::memory_order_acquire) > write.entry); } static GpuSplit instance; @@ -473,14 +715,19 @@ class GpuSplit { static inline thread_local bool on_gpu_thread{}; void (*idle_hook)(void*){}; void* idle_argument{}; + void (*tick_hook)(void*){}; + void* tick_argument{}; alignas(64) std::atomic hook_requested{}; + std::atomic tick_requested{}; std::atomic back_running{}; - std::atomic back_quiet{true}; // 128 bytes: x86 spatial prefetchers pull 64-byte lines in pairs. back_waiting and wake are // always touched together by both threads, so they share a block. alignas(128) std::atomic published{}; alignas(128) std::atomic consumed{}; + /// One past the last entry (EntryTag) whose fenced writes landed (NoteFencedWrite): the + /// fencing thread writes it, the front reads it. + alignas(128) std::atomic fenced_landed{}; /// A position the front waits for (WaitFor): the back stores `consumed` once past it. alignas(128) std::atomic wait_target{~u64{0}}; alignas(128) std::atomic back_waiting{}; @@ -488,10 +735,12 @@ class GpuSplit { /// The front sleeps in Backoff (#161: instead of yielding). alignas(128) std::atomic front_waiting{}; std::atomic front_wake{}; - /// The back's position when it last ran out of work, stored before it waits: while - /// it equals published_local, the back waits for the words the front holds (Commit). Port of - /// the publish part of 62db242b34 (jpkto, PR #805). + /// The back's position when it last ran out of work, stored before it waits for more: while + /// it equals published_local, the back waits for the words the front holds (Commit). The back + /// writes it once per wait, the front reads it at every Commit. alignas(128) std::atomic back_idle_at{~u64{0}}; + /// TEST (profile_publishes): when the back last ran out of work, 0 while it runs. + alignas(128) std::atomic back_idle_since{}; std::jthread back_thread; }; diff --git a/src/video_core/gpu_thread.cpp b/src/video_core/gpu_thread.cpp index 00d931cb53..225b700919 100644 --- a/src/video_core/gpu_thread.cpp +++ b/src/video_core/gpu_thread.cpp @@ -22,9 +22,13 @@ #include "core/frontend/graphics_context.h" #include "video_core/control/scheduler.h" #include "video_core/dma_pusher.h" +#include "video_core/front_qmd.h" #include "video_core/gpu.h" #include "video_core/gpu_thread.h" #include "video_core/gpu_split.h" +#include "video_core/gpu_wait_stats.h" +#include "video_core/nx2_drain_check.h" +#include "video_core/nx2_list_stats.h" #include "video_core/nx2_profile.h" #include "video_core/host1x/host1x.h" #include "video_core/renderer_base.h" @@ -77,12 +81,13 @@ void ThreadManager::StartThread(VideoCore::RendererBase& renderer, Core::Fronten auto last_progress = std::chrono::steady_clock::now(); namespace Profile = VideoCommon::Nx2Profile; auto& split = VideoCommon::GpuSplit::Get(); + namespace WaitStats = VideoCommon::GpuWaitStats; while (!stop_token.stop_requested()) { Profile::Report(); split.MaybeStart(system); - if (VideoCommon::g_acquire_counting) { - split.ReportAcquire(); - } + WaitStats::MaybeReport(split.Active()); + VideoCommon::Nx2DrainCheck::MaybeReport(); + VideoCommon::Nx2AcquireStats::MaybeReport(); if (scheduler.HasPendingChannels()) { // Suspended channels wait on semaphores the CPU releases: poll them meanwhile. bool resumed = false; @@ -93,17 +98,27 @@ void ThreadManager::StartThread(VideoCore::RendererBase& renderer, Core::Fronten if (resumed) { last_progress = std::chrono::steady_clock::now(); } + if (split.Active()) { + // The back may have run everything sent meanwhile. + VideoCommon::FrontQmdView::Get().MaybeTakeOver( + split, VideoCommon::Nx2AcquireStats::QmdTakeOver::BetweenCommands); + } if (!state.queue.TryPop(next)) { if (!scheduler.HasPendingChannels()) { continue; } + // Words sent since the last publish would wait out the sleep below while the + // back runs out of work: hand them over first. + if (split.Active()) { + split.Publish(); + } // The CPU usually releases the semaphore soon after: poll for half a // millisecond, then in short sleeps. Ocarina of Time waits here ~330 times a // second and its present waits for the work behind them: with 1 ms sleeps the // present waited ~17 ms a frame in Kokiri village, with 50 us ~1 ms. A channel // still waiting after 2 s (menus, loading) goes back to 1 ms sleeps. Profile::Scope scope{Profile::Zone::PendingSleep}; - split.SettleIdleBack(); + const WaitStats::IdleScope idle{WaitStats::g_front_poll_ns}; const auto waited = std::chrono::steady_clock::now() - last_progress; if (waited < std::chrono::microseconds{500}) { std::this_thread::yield(); @@ -122,21 +137,30 @@ void ThreadManager::StartThread(VideoCore::RendererBase& renderer, Core::Fronten } } else { Profile::Scope scope{Profile::Zone::Idle}; + const WaitStats::IdleScope idle{WaitStats::g_front_idle_ns}; state.queue.PopWait(next, stop_token); } if (stop_token.stop_requested()) { break; } if (split.Active()) { + // Between commands the back may have run out of work, and everything sent. + split.PublishSoon(); + VideoCommon::FrontQmdView::Get().MaybeTakeOver( + split, VideoCommon::Nx2AcquireStats::QmdTakeOver::BetweenCommands); // Only command lists are parsed here; everything else runs on the back thread // in order, and so does the signal that the command ran. auto* const rasterizer_ptr = renderer.ReadRasterizer(); if (auto* submit_list = std::get_if(&next.data)) { Profile::Scope scope{Profile::Zone::Push}; + Tegra::Nx2ListStats::NoteListQueued(submit_list->channel, -1); scheduler.Push(system.GPU(), submit_list->channel, std::move(submit_list->entries)); } else if (std::holds_alternative(next.data)) { + // The sync operations (the back may run them between any two entries too, + // GpuSplit::RequestTick): flushes write guest memory where GPU work wrote. // TickWork ends fast draw reuse itself when it runs a request. split.Defer([gpu = &system.GPU()] { gpu->TickWork(); }, + VideoCommon::GpuSplit::Writes::Caches, VideoCommon::GpuSplit::FastDrawNeutral); } else if (auto* run = std::get_if(&next.data)) { if (run->before_later_reads) { @@ -145,27 +169,33 @@ void ThreadManager::StartThread(VideoCore::RendererBase& renderer, Core::Fronten split.Drain(VideoCommon::DrainSite::Map); } else { split.DeferFunction(std::move(run->func)); + split.TagUnplaced(VideoCommon::UnplacedKind::RunCommand); // TEST } } else if (const auto* flush = std::get_if(&next.data)) { - split.Defer([rasterizer_ptr, addr = flush->addr, size = flush->size] { - rasterizer_ptr->FlushRegion(addr, size); - }); + // Writes guest memory with what the caches hold of it. + split.Defer([rasterizer_ptr, addr = flush->addr, + size = flush->size] { rasterizer_ptr->FlushRegion(addr, size); }, + VideoCommon::GpuSplit::Writes::Caches); } else if (const auto* invalidate = std::get_if(&next.data)) { - split.Defer([rasterizer_ptr, addr = invalidate->addr, size = invalidate->size] { - rasterizer_ptr->OnCacheInvalidation(addr, size); - }); + split.Defer( + [rasterizer_ptr, addr = invalidate->addr, size = invalidate->size] { + rasterizer_ptr->OnCacheInvalidation(addr, size); + }, + VideoCommon::GpuSplit::Writes::Noted); } else { ASSERT(false); } - split.Defer([state_ptr = &state, fence = next.fence, block = next.block] { - SignalCommand(*state_ptr, fence, block); - }, VideoCommon::GpuSplit::FastDrawNeutral); + split.Defer([state_ptr = &state, fence = next.fence, + block = next.block] { SignalCommand(*state_ptr, fence, block); }, + VideoCommon::GpuSplit::Writes::Noted, + VideoCommon::GpuSplit::FastDrawNeutral); split.Publish(); last_progress = std::chrono::steady_clock::now(); continue; } if (auto* submit_list = std::get_if(&next.data)) { Profile::Scope scope{Profile::Zone::Push}; + Tegra::Nx2ListStats::NoteListQueued(submit_list->channel, -1); scheduler.Push(system.GPU(), submit_list->channel, std::move(submit_list->entries)); } else if (std::holds_alternative(next.data)) { Profile::Scope scope{Profile::Zone::Tick}; @@ -189,6 +219,7 @@ void ThreadManager::StartThread(VideoCore::RendererBase& renderer, Core::Fronten } void ThreadManager::SubmitList(s32 channel, Tegra::CommandList&& entries, bool is_async) { + Tegra::Nx2ListStats::NoteListQueued(channel, 1); PushCommand(SubmitListCommand(channel, std::move(entries)), false, is_async); } diff --git a/src/video_core/gpu_wait_stats.h b/src/video_core/gpu_wait_stats.h new file mode 100644 index 0000000000..49c02bb5da --- /dev/null +++ b/src/video_core/gpu_wait_stats.h @@ -0,0 +1,612 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "common/common_types.h" +#include "common/cpu_features.h" +#include "common/logging.h" +#include "core/hardware_properties.h" +#include "video_core/rasterizer_download_area.h" + +// TEST: how long guest threads wait for the GPU. Always on: one line every 10 s ("GPU waits"). +// +// A guest read of GPU-written memory that GetFlushArea can't serve (reactive flushing) queues a +// flush on the GPU thread and waits for it. Each wait is timed from the request to the wake-up and +// split into: queue (until the GPU thread starts the flush), flush (applying the collected CPU +// writes, then the downloads; "host GPU" is the time in Vulkan's Scheduler::Finish, waiting for +// the host GPU), wake (until the waiting thread runs again) and the copies (the waiting thread's +// own wait for the downloads the flush submitted, and writing them to guest memory: FinishRead). +// Also counted: guest reads and writes that take the rasterizer path, by core, with the time of +// their cache lookups (GetFlushArea, OnCPUWrite: both take the cache locks the GPU thread holds +// while it draws); the other sync operations guest threads wait for; the GPU thread's (and the +// split's back thread's) idle time; every Scheduler::Finish and every wait for pending query +// results (FlushQueryResults); the time between presents; and, for the pages that waited longest, +// the GPU writes that marked them. +namespace VideoCommon::GpuWaitStats { + +/// One slot per guest core, the last one for every other thread. +inline constexpr size_t CoreSlots = Core::Hardware::NUM_CPU_CORES + 1; +/// Pages whose GPU writers are recorded (the longest waits of the last window). +inline constexpr size_t WatchSlots = 4; +/// Distinct flush areas totaled per window. +inline constexpr size_t MaxAreas = 64; +inline constexpr u64 WindowNs = 10'000'000'000ULL; +inline constexpr u64 PageSize = 0x1000; + +/// Steady nanoseconds: the time stamp counter where the wall clock reads it directly, else +/// steady_clock (as Nx2Profile::Now). Cheap enough for the per-read counters. +inline u64 Now() { + return static_cast(Common::g_wall_clock.GetTimeNS().count()); +} + +inline void AtomicMax(std::atomic& value, u64 candidate) { + u64 current = value.load(std::memory_order_relaxed); + while (candidate > current && + !value.compare_exchange_weak(current, candidate, std::memory_order_relaxed)) { + } +} + +/// Increments a counter only one thread writes (no locked instruction). +inline void BumpOwned(std::atomic& value, u64 amount = 1) { + value.store(value.load(std::memory_order_relaxed) + amount, std::memory_order_relaxed); +} + +/// What marked buffer memory GPU-written (BufferCache::MarkWrittenBuffer and DMACopy). +enum class WriteSite : u8 { + Other, + GraphicsStorage, + GraphicsTexel, + ComputeStorage, + ComputeTexel, + MultiRangeStorage, + TransformFeedback, + ImageToBuffer, ///< ObtainBuffer with MarkAsWritten: DMA copies of images into buffers + DmaCopy, + Count, +}; + +inline constexpr std::array(WriteSite::Count)> write_site_names{ + "other", + "graphics storage", + "graphics texel buffer", + "compute storage", + "compute texel buffer", + "multi-range storage", + "transform feedback", + "image to buffer", + "DMA copy", +}; + +inline constexpr std::array source_names{"other", "texture", "buffer"}; + +/// Sync operations other than reactive flushes that a guest thread waits for. +enum class Site : u8 { + Invalidate, ///< A CPU write over GPU-written buffer memory (InvalidateRegion, preserved) + Idle, ///< GPU::WaitForIdle + Count, +}; + +/// Filled by the GPU thread while the guest thread waits for it. +struct FlushTiming { + u64 start{}; ///< The GPU thread started the flush + u64 applied{}; ///< It applied the CPU writes collected so far + u64 end{}; + u64 host_ns{}; ///< Of the flush: time in Scheduler::Finish + u64 finishes{}; + u64 bytes{}; ///< Downloaded by the flush + /// The reading thread, after the flush: waiting for its downloads and writing them to guest + /// memory (RasterizerInterface::FinishRead) + u64 finish_ns{}; + bool in_flight{}; ///< No flush: only waited for an earlier read's downloads +}; + +struct HostWaits { + u64 ns{}; + u64 count{}; +}; + +/// This thread's Scheduler::Finish totals, so a flush can tell its own host waits. +inline thread_local HostWaits t_host_waits{}; +inline std::atomic g_host_wait_ns{}; +inline std::atomic g_host_waits{}; + +/// Scheduler::Finish took `ns` (submit, then wait for the host GPU). +inline void AddHostWait(u64 ns) { + t_host_waits.ns += ns; + ++t_host_waits.count; + g_host_wait_ns.fetch_add(ns, std::memory_order_relaxed); + g_host_waits.fetch_add(1, std::memory_order_relaxed); +} + +/// Bytes this thread's buffer and texture cache downloads copied back (read as a difference by a +/// reactive flush, like t_host_waits). +inline thread_local u64 t_download_bytes{}; + +inline void AddDownloadBytes(u64 bytes) { + t_download_bytes += bytes; +} + +/// RasterizerVulkan::FlushQueryResults waited `ns` for pending query results (macro DMA reads of +/// report regions: the GPU thread waits for the host GPU to reach a new fence). +inline std::atomic g_query_wait_ns{}; +inline std::atomic g_query_waits{}; + +inline void AddQueryWait(u64 ns) { + g_query_wait_ns.fetch_add(ns, std::memory_order_relaxed); + g_query_waits.fetch_add(1, std::memory_order_relaxed); +} + +struct alignas(64) CoreCounter { + std::atomic value{}; +}; +/// Guest reads that took the rasterizer path (Memory HandleRasterizerDownload), by core. +inline std::array g_rasterizer_reads{}; + +inline void CountRasterizerRead(size_t core) { + if (core < CoreSlots - 1) { + BumpOwned(g_rasterizer_reads[core].value); + } else { + g_rasterizer_reads[CoreSlots - 1].value.fetch_add(1, std::memory_order_relaxed); + } +} + +/// Guest writes that took the rasterizer path (Memory HandleRasterizerWrite), and their +/// GPU::OnCPUWrite calls (the buffer and texture cache locks) with time, by core. As in Memory, +/// the last core's slot also takes the other threads, under Memory's guard: one writer at a time. +inline std::array g_rasterizer_writes{}; +inline std::array g_write_checks{}; +inline std::array g_write_check_ns{}; + +inline void CountRasterizerWrite(size_t core) { + if (core < g_rasterizer_writes.size()) { + BumpOwned(g_rasterizer_writes[core].value); + } +} + +inline void AddWriteCheck(size_t core, u64 ns) { + if (core < g_write_checks.size()) { + BumpOwned(g_write_checks[core].value); + BumpOwned(g_write_check_ns[core].value, ns); + } +} + +/// GPU::OnCPURead's GetFlushArea calls (served or not) and their time (two cache locks), by core +/// (GetCurrentHostThreadID; the last slot takes every other thread). +inline std::array g_area_checks{}; +inline std::array g_area_check_ns{}; + +inline void AddAreaCheck(size_t core, u64 ns) { + if (core < CoreSlots - 1) { + BumpOwned(g_area_checks[core].value); + BumpOwned(g_area_check_ns[core].value, ns); + } else { + g_area_checks[CoreSlots - 1].value.fetch_add(1, std::memory_order_relaxed); + g_area_check_ns[CoreSlots - 1].value.fetch_add(ns, std::memory_order_relaxed); + } +} + +/// Idle time of the GPU thread (waiting for commands; polling suspended channels) and of the +/// split's back thread (waiting for the front). +inline std::atomic g_front_idle_ns{}; +inline std::atomic g_front_poll_ns{}; +inline std::atomic g_back_idle_ns{}; + +class IdleScope { +public: + explicit IdleScope(std::atomic& sink_) : sink{sink_}, start{Now()} {} + ~IdleScope() { + sink.fetch_add(Now() - start, std::memory_order_relaxed); + } + IdleScope(const IdleScope&) = delete; + IdleScope& operator=(const IdleScope&) = delete; + +private: + std::atomic& sink; + u64 start; +}; + +struct SyncTotals { + std::atomic count{}; + std::atomic ns{}; +}; +inline std::array(Site::Count)> g_syncs{}; + +inline void RecordSync(Site site, u64 ns) { + auto& totals = g_syncs[static_cast(site)]; + totals.count.fetch_add(1, std::memory_order_relaxed); + totals.ns.fetch_add(ns, std::memory_order_relaxed); +} + +/// Time between presents (nx2:vi present). +inline std::atomic g_last_present{}; +inline std::atomic g_presents{}; +inline std::atomic g_present_ns{}; +inline std::atomic g_present_max_ns{}; + +inline void NotePresent() { + const u64 now = Now(); + const u64 last = g_last_present.exchange(now, std::memory_order_relaxed); + if (last == 0 || now <= last) { + return; + } + g_presents.fetch_add(1, std::memory_order_relaxed); + g_present_ns.fetch_add(now - last, std::memory_order_relaxed); + AtomicMax(g_present_max_ns, now - last); +} + +/// GPU writes over a watched page: written by the buffer cache under its lock, read by the report. +struct alignas(64) Watch { + std::atomic page{}; ///< 0: unused + std::atomic writes{}; + std::atomic last_start{}; + std::atomic last_size{}; + std::atomic largest{}; + std::atomic last_site{}; +}; +inline std::array g_watches{}; +inline std::atomic g_watching{}; + +/// The buffer cache marked [addr, addr + size) GPU-written. +inline void NoteGpuWrite(u64 addr, u64 size, WriteSite site) { + if (g_watching.load(std::memory_order_relaxed) == 0) { + return; + } + for (Watch& watch : g_watches) { + const u64 page = watch.page.load(std::memory_order_relaxed); + if (page == 0 || page >= addr + size || addr >= page + PageSize) { + continue; + } + BumpOwned(watch.writes); + watch.last_start.store(addr, std::memory_order_relaxed); + watch.last_size.store(size, std::memory_order_relaxed); + watch.last_site.store(static_cast(site), std::memory_order_relaxed); + if (size > watch.largest.load(std::memory_order_relaxed)) { + watch.largest.store(size, std::memory_order_relaxed); + } + } +} + +struct AreaTotals { + u64 start{}; + u64 size{}; + u64 count{}; + u64 wait_ns{}; + u64 max_ns{}; + VideoCore::DownloadAreaSource source{}; +}; + +struct ReactiveTotals { + u64 count{}; + u64 wait_ns{}; + u64 max_ns{}; + u64 queue_ns{}; + u64 apply_ns{}; + u64 flush_ns{}; + u64 host_ns{}; + u64 finishes{}; + u64 without_finish{}; + u64 bytes{}; + u64 max_bytes{}; + u64 wake_ns{}; + u64 finish_ns{}; + u64 in_flight{}; + u64 other_areas{}; ///< Flushes of areas past MaxAreas + std::array count_by_source{}; + std::array ns_by_source{}; + std::array ns_by_core{}; + std::vector areas; +}; +inline std::mutex g_reactive_mutex; +inline ReactiveTotals g_reactive; + +/// A guest thread (`core`, GetCurrentHostThreadID) waited from `request` to `woke` for the flush +/// of `area`, which the GPU thread ran as `timing` says. +inline void RecordReactive(const VideoCore::RasterizerDownloadArea& area, size_t core, u64 request, + const FlushTiming& timing, u64 woke) { + const u64 wait = woke - request; + const size_t source = (std::min)(static_cast(area.source), 2); + std::scoped_lock lock{g_reactive_mutex}; + ReactiveTotals& totals = g_reactive; + ++totals.count; + totals.wait_ns += wait; + totals.max_ns = (std::max)(totals.max_ns, wait); + if (timing.start >= request && timing.end >= timing.start && + woke >= timing.end + timing.finish_ns) { + totals.queue_ns += timing.start - request; + totals.apply_ns += timing.applied - timing.start; + totals.flush_ns += timing.end - timing.start; + totals.wake_ns += woke - timing.end - timing.finish_ns; + } + totals.finish_ns += timing.finish_ns; + totals.in_flight += timing.in_flight ? 1 : 0; + totals.host_ns += timing.host_ns; + totals.finishes += timing.finishes; + totals.without_finish += timing.finishes == 0 ? 1 : 0; + totals.bytes += timing.bytes; + totals.max_bytes = (std::max)(totals.max_bytes, timing.bytes); + ++totals.count_by_source[source]; + totals.ns_by_source[source] += wait; + totals.ns_by_core[(std::min)(core, CoreSlots - 1)] += wait; + const auto it = + std::find_if(totals.areas.begin(), totals.areas.end(), [&area](const AreaTotals& entry) { + return entry.start == area.start_address; + }); + AreaTotals* entry = it != totals.areas.end() ? &*it : nullptr; + if (entry == nullptr) { + if (totals.areas.size() >= MaxAreas) { + ++totals.other_areas; + return; + } + entry = &totals.areas.emplace_back(); + entry->start = area.start_address; + entry->source = area.source; + } + entry->size = (std::max)(entry->size, area.end_address - area.start_address); + ++entry->count; + entry->wait_ns += wait; + entry->max_ns = (std::max)(entry->max_ns, wait); +} + +/// Report state: the cumulative counters at the start of the window (GPU thread only). +struct ReportState { + u64 window_start{}; + std::array reads{}; + std::array writes{}; + std::array write_checks{}; + std::array write_check_ns{}; + u64 area_checks{}; + u64 area_check_ns{}; + u64 front_idle{}; + u64 front_poll{}; + u64 back_idle{}; + u64 host_waits{}; + u64 host_wait_ns{}; + u64 query_waits{}; + u64 query_wait_ns{}; + std::array(Site::Count)> sync_count{}; + std::array(Site::Count)> sync_ns{}; + u64 presents{}; + u64 present_ns{}; +}; +inline ReportState g_report; + +/// Takes the cumulative counters' current values; returns the old ones. +inline ReportState Snapshot(u64 now) { + ReportState state; + state.window_start = now; + for (size_t i = 0; i < CoreSlots; ++i) { + state.reads[i] = g_rasterizer_reads[i].value.load(std::memory_order_relaxed); + } + for (size_t i = 0; i < state.writes.size(); ++i) { + state.writes[i] = g_rasterizer_writes[i].value.load(std::memory_order_relaxed); + state.write_checks[i] = g_write_checks[i].value.load(std::memory_order_relaxed); + state.write_check_ns[i] = g_write_check_ns[i].value.load(std::memory_order_relaxed); + } + for (size_t i = 0; i < CoreSlots; ++i) { + state.area_checks += g_area_checks[i].value.load(std::memory_order_relaxed); + state.area_check_ns += g_area_check_ns[i].value.load(std::memory_order_relaxed); + } + state.front_idle = g_front_idle_ns.load(std::memory_order_relaxed); + state.front_poll = g_front_poll_ns.load(std::memory_order_relaxed); + state.back_idle = g_back_idle_ns.load(std::memory_order_relaxed); + state.host_waits = g_host_waits.load(std::memory_order_relaxed); + state.host_wait_ns = g_host_wait_ns.load(std::memory_order_relaxed); + state.query_waits = g_query_waits.load(std::memory_order_relaxed); + state.query_wait_ns = g_query_wait_ns.load(std::memory_order_relaxed); + for (size_t i = 0; i < state.sync_count.size(); ++i) { + state.sync_count[i] = g_syncs[i].count.load(std::memory_order_relaxed); + state.sync_ns[i] = g_syncs[i].ns.load(std::memory_order_relaxed); + } + state.presents = g_presents.load(std::memory_order_relaxed); + state.present_ns = g_present_ns.load(std::memory_order_relaxed); + return std::exchange(g_report, state); +} + +inline double Ms(u64 ns) { + return static_cast(ns) / 1e6; +} + +/// Logs the window's totals every 10 s. Called by the GPU thread between commands, so its own +/// idle time is complete; `split_active` says whether the back thread's counts apply. +inline void MaybeReport(bool split_active) { + const u64 now = Now(); + if (g_report.window_start == 0) { + (void)Snapshot(now); + return; + } + if (now - g_report.window_start < WindowNs) { + return; + } + const ReportState old = Snapshot(now); + const ReportState& cur = g_report; + const u64 window = now - old.window_start; + ReactiveTotals reactive; + { + std::scoped_lock lock{g_reactive_mutex}; + reactive = std::move(g_reactive); + g_reactive = {}; + } + const u64 presents = cur.presents - old.presents; + const u64 present_ns = cur.present_ns - old.present_ns; + const u64 present_max = g_present_max_ns.exchange(0, std::memory_order_relaxed); + + std::string line = fmt::format("GPU waits in {:.1f} s:", static_cast(window) / 1e9); + if (presents != 0) { + line += fmt::format(" {} frames ({:.1f}/s, {:.2f} ms each, max {:.2f} ms) |", presents, + static_cast(presents) * 1e9 / static_cast(window), + Ms(present_ns) / static_cast(presents), Ms(present_max)); + } else { + line += " no presents |"; + } + line += + fmt::format(" reactive flushes {} waited {:.1f} ms ({:.1f}% of the time", reactive.count, + Ms(reactive.wait_ns), + static_cast(reactive.wait_ns) * 100.0 / static_cast(window)); + if (presents != 0) { + line += fmt::format(", {:.2f} ms and {:.2f} flushes a frame", + Ms(reactive.wait_ns) / static_cast(presents), + static_cast(reactive.count) / static_cast(presents)); + } + line += fmt::format("), max {:.2f} ms", Ms(reactive.max_ns)); + // Ends a list of ", "-separated items: drops the separator after the last one. + const auto end_list = [&line] { + if (line.size() >= 1 && line.back() == ',') { + line.pop_back(); + } + }; + if (reactive.count != 0) { + line += "; by cache:"; + for (size_t i = 0; i < source_names.size(); ++i) { + if (reactive.count_by_source[i] != 0) { + line += fmt::format(" {} {} {:.1f} ms,", source_names[i], + reactive.count_by_source[i], Ms(reactive.ns_by_source[i])); + } + } + end_list(); + line += fmt::format(" | queue {:.1f} ms, flush {:.1f} ms (CPU writes {:.1f}, host GPU " + "{:.1f} in {} finishes, {} flushes without one; downloaded {} KiB, " + "at most {} KiB), wake {:.1f} ms, then copies on the reading thread " + "{:.1f} ms ({} reads only waited for an earlier read's) | by core:", + Ms(reactive.queue_ns), Ms(reactive.flush_ns), Ms(reactive.apply_ns), + Ms(reactive.host_ns), reactive.finishes, reactive.without_finish, + reactive.bytes / 1024, reactive.max_bytes / 1024, Ms(reactive.wake_ns), + Ms(reactive.finish_ns), reactive.in_flight); + for (size_t i = 0; i < CoreSlots; ++i) { + if (reactive.ns_by_core[i] == 0) { + continue; + } + line += i + 1 < CoreSlots + ? fmt::format(" {} {:.1f} ms,", i, Ms(reactive.ns_by_core[i])) + : fmt::format(" other {:.1f} ms,", Ms(reactive.ns_by_core[i])); + } + end_list(); + } + // The longest waits, with the GPU writes seen over them (watched since the last report). + std::sort(reactive.areas.begin(), reactive.areas.end(), + [](const AreaTotals& a, const AreaTotals& b) { return a.wait_ns > b.wait_ns; }); + line += " | areas:"; + const size_t shown = (std::min)(reactive.areas.size(), WatchSlots); + if (shown == 0) { + line += " none"; + } + for (size_t i = 0; i < shown; ++i) { + const AreaTotals& area = reactive.areas[i]; + line += fmt::format(" {:#x}+{:#x} {} {}x {:.1f} ms (max {:.2f})", area.start, area.size, + source_names[(std::min)(static_cast(area.source), 2)], + area.count, Ms(area.wait_ns), Ms(area.max_ns)); + for (const Watch& watch : g_watches) { + if (watch.page.load(std::memory_order_relaxed) != area.start) { + continue; + } + const u64 writes = watch.writes.load(std::memory_order_relaxed); + if (writes == 0) { + line += " no GPU writes seen"; + } else { + const u32 site = (std::min)(watch.last_site.load(std::memory_order_relaxed), + static_cast(WriteSite::Count) - 1); + line += fmt::format(" GPU-written {}x, last by {} {:#x}+{:#x} (largest {:#x})", + writes, write_site_names[site], + watch.last_start.load(std::memory_order_relaxed), + watch.last_size.load(std::memory_order_relaxed), + watch.largest.load(std::memory_order_relaxed)); + } + } + line += ','; + } + if (reactive.areas.size() > shown || reactive.other_areas != 0) { + line += + fmt::format(" {} more areas,", reactive.areas.size() - shown + reactive.other_areas); + } + end_list(); + // Watch the longest waits' buffer pages during the next window. + g_watching.store(0, std::memory_order_relaxed); + u32 watching = 0; + for (size_t i = 0; i < WatchSlots; ++i) { + Watch& watch = g_watches[i]; + u64 page = 0; + if (i < shown && reactive.areas[i].source == VideoCore::DownloadAreaSource::BufferCache) { + page = reactive.areas[i].start; + } + watch.page.store(0, std::memory_order_relaxed); + watch.writes.store(0, std::memory_order_relaxed); + watch.last_start.store(0, std::memory_order_relaxed); + watch.last_size.store(0, std::memory_order_relaxed); + watch.largest.store(0, std::memory_order_relaxed); + watch.last_site.store(0, std::memory_order_relaxed); + watch.page.store(page, std::memory_order_relaxed); + watching += page != 0 ? 1 : 0; + } + g_watching.store(watching, std::memory_order_relaxed); + + line += " | other waits:"; + constexpr std::array(Site::Count)> site_names{"invalidate", + "idle"}; + for (size_t i = 0; i < site_names.size(); ++i) { + line += + fmt::format(" {} {}x {:.1f} ms,", site_names[i], cur.sync_count[i] - old.sync_count[i], + Ms(cur.sync_ns[i] - old.sync_ns[i])); + } + end_list(); + line += " | rasterizer-path reads by core:"; + u64 all_reads = 0; + for (size_t i = 0; i < CoreSlots; ++i) { + const u64 reads = cur.reads[i] - old.reads[i]; + all_reads += reads; + if (reads != 0) { + line += i + 1 < CoreSlots ? fmt::format(" {} {},", i, reads) + : fmt::format(" other {},", reads); + } + } + if (all_reads == 0) { + line += " none"; + } + end_list(); + line += fmt::format("; flush area checks {} in {:.1f} ms", cur.area_checks - old.area_checks, + Ms(cur.area_check_ns - old.area_check_ns)); + line += " | rasterizer-path writes by core (OnCPUWrite calls, time):"; + u64 all_writes = 0; + for (size_t i = 0; i < cur.writes.size(); ++i) { + const u64 writes = cur.writes[i] - old.writes[i]; + all_writes += writes; + if (writes != 0) { + line += fmt::format(" {} {} ({}, {:.1f} ms),", i, writes, + cur.write_checks[i] - old.write_checks[i], + Ms(cur.write_check_ns[i] - old.write_check_ns[i])); + } + } + if (all_writes == 0) { + line += " none"; + } + end_list(); + const auto percent = [window](u64 ns) { + return static_cast(ns) * 100.0 / static_cast(window); + }; + const u64 front_idle = cur.front_idle - old.front_idle; + const u64 front_poll = cur.front_poll - old.front_poll; + line += fmt::format(" | GPU thread busy {:.1f}% (idle {:.1f}%, polling suspended channels " + "{:.1f}%)", + 100.0 - percent(front_idle + front_poll), percent(front_idle), + percent(front_poll)); + if (split_active) { + line += fmt::format(", back thread busy {:.1f}%", + 100.0 - percent(cur.back_idle - old.back_idle)); + } + line += + fmt::format(" | Scheduler::Finish {}x {:.1f} ms; query result waits {}x {:.1f} ms", + cur.host_waits - old.host_waits, Ms(cur.host_wait_ns - old.host_wait_ns), + cur.query_waits - old.query_waits, Ms(cur.query_wait_ns - old.query_wait_ns)); + LOG_INFO(HW_GPU, "{}", line); +} + +} // namespace VideoCommon::GpuWaitStats diff --git a/src/video_core/host_shaders/CMakeLists.txt b/src/video_core/host_shaders/CMakeLists.txt index d6a480a271..6a9bb494f3 100644 --- a/src/video_core/host_shaders/CMakeLists.txt +++ b/src/video_core/host_shaders/CMakeLists.txt @@ -69,6 +69,7 @@ set(SHADER_FILES ${CMAKE_CURRENT_SOURCE_DIR}/queries_prefix_scan_sum.comp ${CMAKE_CURRENT_SOURCE_DIR}/queries_prefix_scan_sum_nosubgroups.comp ${CMAKE_CURRENT_SOURCE_DIR}/resolve_conditional_render.comp + ${CMAKE_CURRENT_SOURCE_DIR}/resolve_query_availability.comp ${CMAKE_CURRENT_SOURCE_DIR}/smaa_edge_detection.vert ${CMAKE_CURRENT_SOURCE_DIR}/smaa_edge_detection.frag ${CMAKE_CURRENT_SOURCE_DIR}/smaa_blending_weight_calculation.vert diff --git a/src/video_core/host_shaders/resolve_query_availability.comp b/src/video_core/host_shaders/resolve_query_availability.comp new file mode 100644 index 0000000000..92ef3256c1 --- /dev/null +++ b/src/video_core/host_shaders/resolve_query_availability.comp @@ -0,0 +1,33 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +// Samples query results copied with VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WITH_AVAILABILITY_BIT +// and no wait: each entry is the 64-bit count followed by a 64-bit availability word. A query +// whose result has not landed yet resolves to 1024 samples (visible) instead of whatever the copy +// wrote for it, which the Vulkan spec leaves undefined. + +#version 450 + +layout(local_size_x = 64) in; + +layout(std430, binding = 0) readonly buffer Copied { + uvec4 copied[]; // count low, count high, availability low, availability high +}; + +layout(std430, binding = 1) writeonly buffer Resolved { + uvec2 resolved[]; // 64-bit count +}; + +layout(push_constant) uniform PushConstants { + uint count; +}; + +void main() { + const uint index = gl_GlobalInvocationID.x; + if (index >= count) { + return; + } + const uvec4 entry = copied[index]; + const bool available = (entry.z | entry.w) != 0u; + resolved[index] = available ? entry.xy : uvec2(1024u, 0u); +} diff --git a/src/video_core/macro.cpp b/src/video_core/macro.cpp index 2313727679..494f653aba 100644 --- a/src/video_core/macro.cpp +++ b/src/video_core/macro.cpp @@ -39,7 +39,6 @@ #include "common/bit_field.h" #include "common/logging.h" #include "video_core/engines/maxwell_3d.h" -#include "video_core/nx2_async_readback.h" #include "video_core/nx2_fast_draw.h" #include "video_core/nx2_profile.h" @@ -1401,9 +1400,12 @@ void MacroEngine::Execute(Core::System& system, Engines::Maxwell3D& maxwell3d, u bool read_memory = false; const auto emit = [&](u32 address, u32 value) { if (address >= 0xe00) return false; - if (split_writes != nullptr && !(*split_writes)[address]) { + if (split_writes != nullptr && !(*split_writes)[address]) [[unlikely]] { ++split_unexpected_writes; } + if (split_record != nullptr) [[unlikely]] { + split_record->emplace_back(address, value); + } if (address == 0x556) { // SET_RENDER_ENABLE_OVERRIDE: 0 use the render-enable condition, 1 always, // 2 never. Log a histogram every 20000 macro writes (pop-in diagnostics). @@ -1452,28 +1454,17 @@ void MacroEngine::Execute(Core::System& system, Engines::Maxwell3D& maxwell3d, u VideoCommon::QueryDma::Trace(memory, maxwell3d.Rasterizer(), method, address, source, read_bytes); } - // A call the split's back runs in its front's place (Maxwell3D::FrontRunOnBack) - // counts its write-out for the async read-back line. - const bool back_call = - split_writes != nullptr && VideoCommon::Nx2Profile::Enabled(); - const u64 write_out_start = back_call ? VideoCore::AsyncReadback::Now() : 0; if (owns_caches) { VideoCommon::QueryDma::FlushPending(memory, maxwell3d.Rasterizer(), source, read_bytes); } - const bool dirty = - memory.IsMemoryDirty(source, read_bytes, VideoCommon::CacheType::BufferCache); - if (dirty) { + if (memory.IsMemoryDirty(source, read_bytes, VideoCommon::CacheType::BufferCache)) { memory.FlushRegion(source, read_bytes, VideoCommon::CacheType::BufferCache); static u64 flushes{}; if (++flushes % 2000 == 1) { LOG_INFO(HW_GPU, "Native MME DMA read flushed GPU-written memory ({} times)", flushes); } } - if (back_call) { - VideoCore::AsyncReadback::CountBackDmaRead( - dirty, VideoCore::AsyncReadback::Now() - write_out_start); - } static u32 dma_logs{}; if (dma_logs < 400) { ++dma_logs; diff --git a/src/video_core/macro.h b/src/video_core/macro.h index d213b86efd..b739d633f7 100644 --- a/src/video_core/macro.h +++ b/src/video_core/macro.h @@ -271,6 +271,8 @@ struct MacroEngine { /// writes to other registers are counted in split_unexpected_writes. const std::bitset<0xe00>* split_writes{}; u64 split_unexpected_writes{}; + /// EDEN_NX2_BACK_MACRO_VERIFY: where such a call records its writes (method, value). + std::vector>* split_record{}; }; struct NativeMacroContext { diff --git a/src/video_core/macro_turing.h b/src/video_core/macro_turing.h index 56af4e9dbe..7223512a17 100644 --- a/src/video_core/macro_turing.h +++ b/src/video_core/macro_turing.h @@ -261,6 +261,12 @@ class TuringMacroInterpreter { std::vector methods; ///< Ascending, each once bool methods_known{true}; bool data_ram{}; + /// The data RAM words DREAD can read and the registers MME_READ can read, ascending, each + /// once; not all of them where `..._known` is false (an address the code computes). + std::vector data_ram_reads; + bool data_ram_reads_known{true}; + std::vector register_reads; + bool register_reads_known{true}; }; static Effects AnalyzeEffects(std::span code, Word start) { @@ -420,7 +426,13 @@ class TuringMacroInterpreter { case 17: case 18: case 19: case 20: case 21: value = both ? known(taken(d.op[lane]) ? ~0U : 0) : unknown; break; - case 22: value = unknown; break; // A register + case 22: // A register + value = unknown; + if (both) + effects.register_reads.push_back((a + b) & 0xffff); + else + effects.register_reads_known = false; + break; case 23: if (imm < 2) break; // Entered, or skipped with a count of 0. Its last instruction may start @@ -453,7 +465,14 @@ class TuringMacroInterpreter { if (!both || !jumps) flow[flow_count++] = {Flow::Next, 0, false}; break; } - case 30: effects.data_ram = true; value = unknown; break; + case 30: + effects.data_ram = true; + value = unknown; + if (x.known) + effects.data_ram_reads.push_back(a); + else + effects.data_ram_reads_known = false; + break; case 31: effects.data_ram = true; break; default: break; // 7: the load barrier } @@ -513,9 +532,11 @@ class TuringMacroInterpreter { } } } - std::ranges::sort(effects.methods); - effects.methods.erase(std::unique(effects.methods.begin(), effects.methods.end()), - effects.methods.end()); + for (std::vector* list : + {&effects.methods, &effects.data_ram_reads, &effects.register_reads}) { + std::ranges::sort(*list); + list->erase(std::unique(list->begin(), list->end()), list->end()); + } return effects; } diff --git a/src/video_core/memory_manager.cpp b/src/video_core/memory_manager.cpp index bc4dde46d9..90f4ef116f 100644 --- a/src/video_core/memory_manager.cpp +++ b/src/video_core/memory_manager.cpp @@ -329,7 +329,7 @@ T MemoryManager::Read(GPUVAddr addr) const { template void MemoryManager::Write(GPUVAddr addr, T data) { // A download for a guest read not yet in guest memory would land over this write later. - if (VideoCore::g_read_downloads.load(std::memory_order_relaxed) != 0) { + if (VideoCore::g_read_downloads.load(std::memory_order_relaxed) != 0) [[unlikely]] { if (const auto address = GpuToCpuAddress(addr); address && rasterizer != nullptr && VideoCore::ReadDownloadsMayCover(*address, sizeof(T))) { @@ -542,7 +542,7 @@ void MemoryManager::WriteBlockImpl(GPUVAddr gpu_dest_addr, const void* src_buffe }; // A download for a guest read not yet in guest memory would land over this write later. const auto finish_reads = [this, unsafe](DAddr dev_addr, std::size_t amount) { - if (VideoCore::g_read_downloads.load(std::memory_order_relaxed) != 0) { + if (VideoCore::g_read_downloads.load(std::memory_order_relaxed) != 0) [[unlikely]] { if (rasterizer != nullptr && VideoCore::ReadDownloadsMayCover(dev_addr, amount)) { if (VideoCore::ReadCheckWrites::Enabled()) { VideoCore::ReadCheckWrites::Note( diff --git a/src/video_core/memory_manager.h b/src/video_core/memory_manager.h index 380ebce967..4478cbf744 100644 --- a/src/video_core/memory_manager.h +++ b/src/video_core/memory_manager.h @@ -48,6 +48,13 @@ class MemoryManager final { // Switch 2: GPU address of the read-only syncpoint shim (u32 per syncpoint); 0 = none. void SetNativeSyncpointShim(GPUVAddr address); static constexpr u32 NumShimSyncpoints = 192; // Tegra::Host1x::SyncpointManager + /// Whether [address, address + size) overlaps the syncpoint shim, whose words Read serves + /// from the guest syncpoints instead of memory. + [[nodiscard]] bool OverlapsSyncpointShim(GPUVAddr address, u64 size) const { + const GPUVAddr shim = native_syncpoint_shim.load(std::memory_order_acquire); + return shim != 0 && address < shim + NumShimSyncpoints * sizeof(u32) && + address + size > shim; + } /// Switch 2 lays out block-linear surfaces with the console's GOB swizzle. void EnableNativeTiling() { diff --git a/src/video_core/native_qmd.h b/src/video_core/native_qmd.h index f32afb363b..ed16358774 100644 --- a/src/video_core/native_qmd.h +++ b/src/video_core/native_qmd.h @@ -5,10 +5,13 @@ #include #include #include +#include #include #include #include #include +#include +#include namespace Tegra { @@ -37,6 +40,52 @@ class NativeQmdScheduler { using Address = std::uint64_t; using Descriptor = std::array; struct RejectShaderDispatch { bool operator()(const Descriptor&) const { return false; } }; + /// A scheduler with a table of its own, not the one the channels share: the split GPU thread's + /// front runs the scheduling ahead of its back to know what the launches release + /// (KeplerCompute). + struct OwnTable {}; + NativeQmdScheduler() = default; + explicit NativeQmdScheduler(OwnTable) : table{&own_states} {} + NativeQmdScheduler(const NativeQmdScheduler&) = delete; + NativeQmdScheduler& operator=(const NativeQmdScheduler&) = delete; + /// Takes the QMD states of `other` over, while nothing runs on either. + void CopyStates(const NativeQmdScheduler& other) { + if (table != other.table) { + *table = *other.table; + } + } + /// TEST (EDEN_NX2_DRAIN_CHECK): whether `other` holds the same QMD states, while nothing + /// runs on either; how many QMDs this one holds. + [[nodiscard]] bool SameStates(const NativeQmdScheduler& other) const { + return *table == *other.table; + } + [[nodiscard]] std::size_t StateCount() const { + return table->size(); + } + /// The split GPU thread's front runs signals it may have to take back, where it can't read a + /// QMD as the back will yet (FrontQmdView): from BeginAttempt on, Signal records what it + /// changes in the table, until CommitAttempt keeps it or RollbackAttempt restores the table. + void BeginAttempt() { + undo.clear(); + attempting = true; + } + void CommitAttempt() { + undo.clear(); + attempting = false; + } + void RollbackAttempt() { + auto& states = *table; + // Newest first: each QMD ends with the state it had before the attempt. + for (auto it = undo.rbegin(); it != undo.rend(); ++it) { + if (it->second) { + states.insert_or_assign(it->first, *it->second); + } else { + states.erase(it->first); + } + } + undo.clear(); + attempting = false; + } // Optional diagnostics: called with "load"/"count"/"defer"/"run" events. std::function trace; // Optional diagnostics: called with the QMD, slot and raw words of a release it skips. @@ -57,6 +106,9 @@ class NativeQmdScheduler { template std::string_view Signal(Address address, Word argument, Read read, Release release, Dispatch dispatch) { struct Pending { Address address; Word action; Word count; }; + auto& states = *table; + // What `read` returns is used before the next read only: a table entry holds no copy. + Descriptor descriptor{}; std::string_view error; std::array pending{}; std::size_t count = 1; @@ -65,16 +117,29 @@ class NativeQmdScheduler { const auto request = pending[i]; if (states.size() >= 4096 && !states.contains(request.address)) { // Finished QMDs with nothing deferred are only history, drop them instead of stalling. + if (attempting) { + for (const auto& [kept_address, kept] : states) { + if (kept.completed && kept.deferred == 0) { + undo.emplace_back(kept_address, kept); + } + } + } std::erase_if(states, [](const auto& entry) { return entry.second.completed && entry.second.deferred == 0; }); if (states.size() >= 4096) return "QMD cache limit"; } + if (attempting) { + const auto found = states.find(request.address); + undo.emplace_back(request.address, found != states.end() + ? std::optional{found->second} + : std::nullopt); + } auto& state = states[request.address]; if (request.action == 3 || request.action == 2) { if (request.action == 3 || !state.loaded) { - if (!read(request.address, state.descriptor)) return "unmapped QMD"; - const auto& qmd = state.descriptor; + if (!read(request.address, descriptor)) return "unmapped QMD"; + const auto& qmd = descriptor; if ((qmd[18] & 0xff) != 0x30 || (qmd[4] & 0x100)) return "unsupported QMD version or queue"; if constexpr (std::is_same_v) { if (qmd[12] && (qmd[13] & 0xffff) && (qmd[14] & 0xffff)) @@ -106,8 +171,8 @@ class NativeQmdScheduler { if (state.dependencies < 0) return "QMD dependency underflow"; // The SDK patches release payloads while a scheduled QMD is // waiting for its remaining dependencies. Fetch them at execution. - if (!read(request.address, state.descriptor)) return "unmapped QMD"; - const auto qmd = state.descriptor; + if (!read(request.address, descriptor)) return "unmapped QMD"; + const auto qmd = descriptor; if ((qmd[18] & 0xff) != 0x30 || (qmd[4] & 0x100)) return "unsupported QMD version or queue"; // A program Eden can't run is skipped, but the QMD still completes: its releases // are what other queues and the CPU wait on (the console never fails a launch). @@ -144,16 +209,22 @@ class NativeQmdScheduler { static constexpr std::array ReleaseSlots{24, 28, 52}; struct State { - Descriptor descriptor{}; std::int32_t dependencies{}; std::int32_t deferred{}; bool loaded{}; bool scheduled{}; bool completed{}; + + bool operator==(const State&) const = default; ///< TEST: SameStates }; // One table for every channel: a QMD's dependency counter is state of the QMD itself, and // title 1 schedules a QMD on one channel while another decrements it (a decrement that // arrived first was kept by the other channel's scheduler and the QMD never launched). - static inline std::unordered_map states; + static inline std::unordered_map shared_states; + std::unordered_map own_states; + std::unordered_map* table{&shared_states}; + /// BeginAttempt: the states the attempt changed, as they were (none: the QMD had no state). + bool attempting{}; + std::vector>> undo; }; } // namespace Tegra diff --git a/src/video_core/nx2.h b/src/video_core/nx2.h index 0913a55641..e152a53671 100644 --- a/src/video_core/nx2.h +++ b/src/video_core/nx2.h @@ -479,46 +479,32 @@ inline u64 Nx2SemaphoreMirror(u64 address) { /// Channels suspended on a semaphore acquire. A console channel polls and sees every release; /// Eden writes releases in bursts when fences signal and resumes waiters only between command /// lists, so an acquire-equal could miss its value. Writes latch the waiters they satisfy. -/// The channels' acquires (Add, Remove, Latched) all run on the GPU thread (the split's front); -/// writes (OnWrite) come from the thread that runs the engines. Latched takes no lock: a waiter -/// polls its own slot, which only its own thread fills and empties. class Nx2SemaphoreWaiters { public: // Puller GpuSemaphoreOperation values. static constexpr u32 AcquireEqual = 0x1; static constexpr u32 AcquireGequal = 0x4; static constexpr u32 AcquireMask = 0x8; - /// One per channel suspended at once. - static constexpr size_t MaxWaiters = 64; + /// TEST (EDEN_NX2_ACQUIRE_STATS): the writer that latched a waiter. + enum class Source : u8 { + Unknown, + PayloadRelease, + QueryResult, + QmdReduction, + QmdMax, + Count, + }; static Nx2SemaphoreWaiters& Get() { static Nx2SemaphoreWaiters waiters; return waiters; } - /// False, and nothing is latched for owner, when MaxWaiters others wait already. bool Add(const void* owner, u64 device_address, u32 value, u32 operation) { std::scoped_lock lk{mutex}; - Slot* free_slot = nullptr; - for (Slot& slot : slots) { - const void* const current = slot.owner.load(std::memory_order_relaxed); - if (current == owner) { - slot.owner.store(nullptr, std::memory_order_relaxed); - count.fetch_sub(1, std::memory_order_relaxed); - } - if ((current == nullptr || current == owner) && free_slot == nullptr) { - free_slot = &slot; - } - } - if (free_slot == nullptr) { - return false; - } - free_slot->address = device_address; - free_slot->value = value; - free_slot->operation = operation; - free_slot->latched.store(false, std::memory_order_relaxed); - free_slot->owner.store(owner, std::memory_order_relaxed); - count.fetch_add(1, std::memory_order_release); + std::erase_if(waiters, [owner](const Waiter& w) { return w.owner == owner; }); + waiters.push_back({owner, device_address, value, operation, false, Source::Unknown}); + count.store(waiters.size(), std::memory_order_release); return true; } @@ -527,65 +513,68 @@ class Nx2SemaphoreWaiters { return; } std::scoped_lock lk{mutex}; - for (Slot& slot : slots) { - if (slot.owner.load(std::memory_order_relaxed) == owner) { - slot.owner.store(nullptr, std::memory_order_relaxed); - count.fetch_sub(1, std::memory_order_relaxed); - } - } + std::erase_if(waiters, [owner](const Waiter& w) { return w.owner == owner; }); + count.store(waiters.size(), std::memory_order_release); } - [[nodiscard]] bool Latched(const void* owner) const { + [[nodiscard]] bool Latched(const void* owner) { if (count.load(std::memory_order_acquire) == 0) { return false; } - for (const Slot& slot : slots) { - if (slot.owner.load(std::memory_order_relaxed) == owner) { - // Acquire: what the engine wrote before the latch is visible after it. - return slot.latched.load(std::memory_order_acquire); + std::scoped_lock lk{mutex}; + return std::ranges::any_of(waiters, + [owner](const Waiter& w) { return w.owner == owner && w.latched; }); + } + + [[nodiscard]] Source LatchSource(const void* owner) { + std::scoped_lock lk{mutex}; + for (const Waiter& w : waiters) { + if (w.owner == owner && w.latched) { + return w.source; } } - return false; + return Source::Unknown; } /// A GPU write of value to the u32 at device_address. - void OnWrite(u64 device_address, u32 value) { + void OnWrite(u64 device_address, u32 value, Source source = Source::Unknown) { if (count.load(std::memory_order_acquire) == 0) { return; } std::scoped_lock lk{mutex}; - for (Slot& slot : slots) { - if (slot.owner.load(std::memory_order_relaxed) != nullptr && - slot.address == device_address && Matches(slot, value)) { - slot.latched.store(true, std::memory_order_release); + for (auto& w : waiters) { + if (w.address == device_address && !w.latched && Matches(w, value)) { + w.latched = true; + w.source = source; } } } private: - struct Slot { - std::atomic owner{}; - u64 address{}; - u32 value{}; - u32 operation{}; - std::atomic latched{}; + struct Waiter { + const void* owner; + u64 address; + u32 value; + u32 operation; + bool latched; + Source source; }; - static bool Matches(const Slot& slot, u32 current) { - switch (slot.operation) { + static bool Matches(const Waiter& w, u32 current) { + switch (w.operation) { case AcquireEqual: - return current == slot.value; + return current == w.value; case AcquireGequal: - return current >= slot.value; + return current >= w.value; case AcquireMask: - return (current & slot.value) != 0; + return (current & w.value) != 0; default: return false; } } - std::mutex mutex; ///< Held to fill, empty and latch slots - std::array slots{}; + std::mutex mutex; + std::vector waiters; std::atomic count{}; }; diff --git a/src/video_core/nx2_async_readback.h b/src/video_core/nx2_async_readback.h deleted file mode 100644 index 47ffb44869..0000000000 --- a/src/video_core/nx2_async_readback.h +++ /dev/null @@ -1,259 +0,0 @@ -// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project -// SPDX-License-Identifier: GPL-3.0-or-later - -#pragma once - -#include -#include -#include -#include -#include - -#include "common/common_types.h" -#include "common/logging.h" - -// Async read-back: a guest CPU read of GPU-written buffer memory (GPU::OnCPURead) has the GPU -// thread submit the downloads and go on. The reading thread waits for that submission and writes -// the bytes to guest memory (BufferCache::FinishReadDownloads). Pages the guest reads after each -// GPU write get their download right behind the write. A macro DMA read on the split's front -// waits only for the queued work that can write its words (Maxwell3D::FrontDmaRead, -// GpuSplit::WaitForWrites); macros whose code reads memory into their FIFO run on the back -// instead (Maxwell3D::FrontRunOnBack). -// -// Diagnostics, off unless set: -// - EDEN_NX2_READ_CHECK=n: 1 in n read downloads (1 checks them all) is compared with a -// synchronous flush's copy taken at the same point of the GPU's work -// (BufferCache::CheckReadDownload). -// - EDEN_NX2_DMA_READ_VERIFY: every macro DMA read made without draining the split drains it -// too and is compared (CheckDmaRead). -namespace VideoCore::AsyncReadback { - -/// EDEN_NX2_READ_CHECK's value: unset, 0 or not a number checks none. -[[nodiscard]] inline u64 ParseReadCheck(const char* value) { - return value == nullptr ? 0 : static_cast(std::strtoull(value, nullptr, 10)); -} - -inline std::atomic g_read_check_every{ParseReadCheck(std::getenv("EDEN_NX2_READ_CHECK"))}; -inline std::atomic g_dma_read_verify{std::getenv("EDEN_NX2_DMA_READ_VERIFY") != nullptr}; - -/// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads -/// nx2_switches.txt, so a switch set only there takes effect from that point on. -inline void ReadSwitch() { - g_read_check_every.store(ParseReadCheck(std::getenv("EDEN_NX2_READ_CHECK")), - std::memory_order_relaxed); - g_dma_read_verify.store(std::getenv("EDEN_NX2_DMA_READ_VERIFY") != nullptr, - std::memory_order_relaxed); -} - -/// EDEN_NX2_READ_CHECK: check 1 in this many read downloads, 0 for none. -[[nodiscard]] inline u64 ReadCheckEvery() { - return g_read_check_every.load(std::memory_order_relaxed); -} - -[[nodiscard]] inline bool DmaReadVerify() { - return g_dma_read_verify.load(std::memory_order_relaxed); -} - -/// Who wrote a read download to guest memory, named by the read check's mismatch lines. -enum class Landing : u8 { - Reader, ///< The guest thread whose read waited for it (GPU::OnCPURead) - GpuSideWrite, ///< A write to guest memory from the GPU's side over its bytes - Completed, ///< Whichever thread found the host GPU had run it, before a read waited - Count, -}; - -inline constexpr std::array(Landing::Count)> landing_names{ - "the reading thread", - "a GPU-side write over it", - "a thread that found it run by the host GPU", -}; - -/// Set on a guest thread while GPU::OnCPURead writes its downloads to guest memory. -inline thread_local bool t_reading{}; - -[[nodiscard]] inline u64 Now() { - return static_cast(std::chrono::duration_cast( - std::chrono::steady_clock::now().time_since_epoch()) - .count()); -} - -enum class Read : u8 { - Async, ///< The GPU thread submitted the downloads, the reading thread waited for them - InFlight, ///< An earlier read's or an early download covered it: no flush, only the wait - Count, -}; - -/// Totals for the per-second line (EDEN_NX2_GPU_PROFILE), from every thread. -struct Counters { - std::array, static_cast(Read::Count)> reads{}; - /// From the read's request to its bytes in guest memory - std::array, static_cast(Read::Count)> read_ns{}; - /// The GPU thread's time in the reads' flushes - std::atomic flush_ns{}; - /// Reading threads whose download the host GPU had not run yet - std::atomic reader_waits{}; - std::atomic reader_wait_ns{}; - std::atomic reader_max_wait_ns{}; - /// Writes from the GPU's side that waited for the host GPU to run a read download under them - std::atomic gpu_side_waits{}; - std::atomic gpu_side_wait_ns{}; - /// Macro DMA reads on the split's front (Maxwell3D::FrontDmaRead), those that waited for its - /// back and those that drained it, and the time they waited - std::atomic dma_reads{}; - std::atomic dma_waited{}; - std::atomic dma_drained{}; - std::atomic dma_wait_ns{}; - /// Landings BufferCache::ClearDownload forced on the GPU thread before a write guest memory - /// doesn't hold yet (a DMA copy, a discarded write), those that waited for the host GPU (also - /// among gpu_side_waits) and the time they waited - std::atomic clear_landings{}; - std::atomic clear_waits{}; - std::atomic clear_wait_ns{}; - /// Macros that read memory, run on the split's back in its front's place - /// (Maxwell3D::FrontRunOnBack): the calls and the back's time in them, their DMA reads, those - /// that found GPU-modified words to flush and the time the reads spent writing the caches out - /// (part of the calls' time) - std::atomic back_macros{}; - std::atomic back_macro_ns{}; - std::atomic back_dma_reads{}; - std::atomic back_dma_flushes{}; - std::atomic back_dma_flush_ns{}; - /// The front's waits for such macros to run: for registers they wrote, for shadow RAM entries - /// they set, and for those sent before the macro code changes - std::atomic back_waits{}; - std::atomic back_wait_ns{}; - std::atomic window_start{}; -}; - -inline Counters g_counters; - -inline void Add(std::atomic& counter, u64 amount) { - counter.fetch_add(amount, std::memory_order_relaxed); -} - -inline void CountRead(Read kind, u64 ns) { - Add(g_counters.reads[static_cast(kind)], 1); - Add(g_counters.read_ns[static_cast(kind)], ns); -} - -/// BufferCache::FinishReadDownloads waited ns for the host GPU. -inline void CountHostWait(u64 ns) { - if (!t_reading) { - Add(g_counters.gpu_side_waits, 1); - Add(g_counters.gpu_side_wait_ns, ns); - return; - } - Add(g_counters.reader_waits, 1); - Add(g_counters.reader_wait_ns, ns); - u64 max = g_counters.reader_max_wait_ns.load(std::memory_order_relaxed); - while (ns > max && !g_counters.reader_max_wait_ns.compare_exchange_weak( - max, ns, std::memory_order_relaxed)) { - } -} - -inline void CountDmaRead(bool waited, bool drained, u64 ns) { - Add(g_counters.dma_reads, 1); - Add(g_counters.dma_waited, waited ? 1 : 0); - Add(g_counters.dma_drained, drained ? 1 : 0); - Add(g_counters.dma_wait_ns, ns); -} - -/// BufferCache::ClearDownload landed read downloads, waiting ns for the host GPU (0: their copies -/// had run). -inline void CountClearLanding(u64 ns) { - Add(g_counters.clear_landings, 1); - Add(g_counters.clear_waits, ns != 0 ? 1 : 0); - Add(g_counters.clear_wait_ns, ns); -} - -/// The split's back ran a macro in its front's place for ns. -inline void CountBackMacro(u64 ns) { - Add(g_counters.back_macros, 1); - Add(g_counters.back_macro_ns, ns); -} - -/// A DMA read of such a macro spent ns writing the caches out, flushing GPU-modified words. -inline void CountBackDmaRead(bool flushed, u64 ns) { - Add(g_counters.back_dma_reads, 1); - Add(g_counters.back_dma_flushes, flushed ? 1 : 0); - Add(g_counters.back_dma_flush_ns, ns); -} - -/// The front waited ns for such a macro to run. -inline void CountBackWait(u64 ns) { - Add(g_counters.back_waits, 1); - Add(g_counters.back_wait_ns, ns); -} - -/// Logs the totals every 10 s and resets them, on whichever thread counting comes first. -inline void MaybeReport() { - auto& c = g_counters; - const u64 now = Now(); - u64 start = c.window_start.load(std::memory_order_relaxed); - if (start == 0) { - c.window_start.compare_exchange_strong(start, now, std::memory_order_relaxed); - return; - } - if (now - start < 10'000'000'000ULL || - !c.window_start.compare_exchange_strong(start, now, std::memory_order_relaxed)) { - return; - } - const double window = static_cast(now - start); - const auto take = [](std::atomic& counter) { - return counter.exchange(0, std::memory_order_relaxed); - }; - const auto per_second = [window](u64 value) { - return static_cast(value) * 1e9 / window; - }; - const auto ms_per_second = [window](u64 ns) { return static_cast(ns) * 1e3 / window; }; - std::array(Read::Count)> reads{}; - std::array(Read::Count)> read_ns{}; - for (size_t i = 0; i < reads.size(); ++i) { - reads[i] = take(c.reads[i]); - read_ns[i] = take(c.read_ns[i]); - } - const u64 flush_ns = take(c.flush_ns); - const u64 reader_waits = take(c.reader_waits); - const u64 reader_wait_ns = take(c.reader_wait_ns); - const u64 reader_max_wait_ns = take(c.reader_max_wait_ns); - const u64 gpu_side_waits = take(c.gpu_side_waits); - const u64 gpu_side_wait_ns = take(c.gpu_side_wait_ns); - const u64 dma_reads = take(c.dma_reads); - const u64 dma_waited = take(c.dma_waited); - const u64 dma_drained = take(c.dma_drained); - const u64 dma_wait_ns = take(c.dma_wait_ns); - const u64 clear_landings = take(c.clear_landings); - const u64 clear_waits = take(c.clear_waits); - const u64 clear_wait_ns = take(c.clear_wait_ns); - const u64 back_macros = take(c.back_macros); - const u64 back_macro_ns = take(c.back_macro_ns); - const u64 back_dma_reads = take(c.back_dma_reads); - const u64 back_dma_flushes = take(c.back_dma_flushes); - const u64 back_dma_flush_ns = take(c.back_dma_flush_ns); - const u64 back_waits = take(c.back_waits); - const u64 back_wait_ns = take(c.back_wait_ns); - constexpr auto async = static_cast(Read::Async); - constexpr auto in_flight = static_cast(Read::InFlight); - LOG_INFO(HW_GPU, - "Async read-back per second: reads {:.0f} async ({:.1f} ms), {:.0f} only waited for a " - "download on its way ({:.1f} ms) | GPU thread in read flushes {:.1f} ms | readers " - "waited on their fence " - "{:.0f} times, {:.1f} ms (longest {:.2f} ms) | GPU-side writes fell back to a host " - "GPU wait {:.0f} times, {:.1f} ms | macro DMA reads {:.0f}, {:.0f} waited for the " - "split's back ({:.1f} ms), {:.0f} drained it | GPU thread landed read downloads before " - "writes guest memory doesn't hold {:.0f} times, {:.0f} waited for the host GPU " - "({:.1f} ms) | split's back ran {:.0f} macros that read memory ({:.1f} ms), their DMA " - "reads {:.0f}, {:.0f} flushed GPU-modified words ({:.1f} ms writing caches out) | " - "front waited for them {:.0f} times ({:.1f} ms)", - per_second(reads[async]), ms_per_second(read_ns[async]), - per_second(reads[in_flight]), ms_per_second(read_ns[in_flight]), - ms_per_second(flush_ns), per_second(reader_waits), ms_per_second(reader_wait_ns), - static_cast(reader_max_wait_ns) / 1e6, per_second(gpu_side_waits), - ms_per_second(gpu_side_wait_ns), per_second(dma_reads), per_second(dma_waited), - ms_per_second(dma_wait_ns), per_second(dma_drained), per_second(clear_landings), - per_second(clear_waits), ms_per_second(clear_wait_ns), per_second(back_macros), - ms_per_second(back_macro_ns), per_second(back_dma_reads), per_second(back_dma_flushes), - ms_per_second(back_dma_flush_ns), per_second(back_waits), ms_per_second(back_wait_ns)); -} - -} // namespace VideoCore::AsyncReadback diff --git a/src/video_core/nx2_drain_check.h b/src/video_core/nx2_drain_check.h new file mode 100644 index 0000000000..755402afc1 --- /dev/null +++ b/src/video_core/nx2_drain_check.h @@ -0,0 +1,908 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "common/common_types.h" +#include "common/logging.h" +#include "video_core/gpu_split.h" +#include "video_core/nx2.h" + +// EDEN_NX2_DRAIN_CHECK: what the split GPU thread's front would see without some of its waits for +// the back, counted without changing what runs and logged every 10 s by the front: +// - command lists on pages no GPU work had written that did not wait for the compute and copy +// work sent after the last host-level wait (DmaPusher::Step): how many of them that work made +// GPU-written, checked on the back at the list's place. It must stay 0; +// - semaphore acquires, and the polls of suspended ones (Puller): whether the value met the +// acquire before the front drained the split and after, by whether the GPU had released a +// semaphore at that address before (any engine, RasterizerInterface::Query with a payload); +// - acquires and polls the semaphore view decides (Puller AskView), verified: the front drains +// and reads the word as before anyway, and counts the decisions the drain contradicts. A pass +// the drained word does not meet must stay 0; such a pass is corrected (the channel waits), and +// a wait the drained word meets ends (counted: a drain would have passed or resumed it); +// - the releases the front noted for the QMD launches it valued (FrontQmdView), against the ones +// the back then ran, compared on the back right after the QMD methods, and the descriptors the +// front read for them (from memory, the signal's inline QMD or a queued one) against the ones +// the back read; the descriptors the front read for signals it valued late (they waited) against +// the ones the back read running them, by their position; whether the back could have run an +// entry the front valued when it read memory for it. Every 512th time, and after every signal +// valued late, the front's whole QMD table against the back's. Wrong ones must stay 0. +namespace VideoCommon::Nx2DrainCheck { + +// A plain flag: the front checks it at every command list and acquire. +inline bool g_enabled = std::getenv("EDEN_NX2_DRAIN_CHECK") != nullptr; + +/// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads +/// nx2_switches.txt, so a switch set only there takes effect from that point on. +inline void ReadSwitch() { + g_enabled = std::getenv("EDEN_NX2_DRAIN_CHECK") != nullptr; +} + +struct State { + std::atomic lists_unwaited{}; + std::atomic lists_written{}; + /// Indexed by poll * 8 + released * 4 + met before the drain * 2 + met after it. + std::array, 16> acquires{}; + /// Passes the semaphore view decided, by poll * 3 + ViewPass, and those the drained word did + /// not meet. + std::array, 6> view_passes{}; + std::array, 6> view_wrong{}; + /// Passes the drained word didn't meet because a fenced release landed during the drain: the + /// drain path itself may read either value then. + std::array, 6> view_raced{}; + /// Acquires the view suspended and polls it kept waiting (by poll), and of those the ones + /// the drained word met. + std::array, 2> view_waits{}; + std::array, 2> view_waits_met{}; + /// QMD methods the front valued and the releases compared for them, those that differed, + /// and methods it could not value (their releases are not compared). + std::atomic qmd_methods{}; + std::atomic qmd_releases{}; + std::atomic qmd_wrong{}; + std::atomic qmd_unvalued{}; + std::atomic qmd_logged{}; + /// Descriptors compared for the valued methods, and those the back read otherwise; QMD tables + /// compared, and those that differed. + std::atomic qmd_reads{}; + std::atomic qmd_reads_wrong{}; + std::atomic qmd_reads_logged{}; + std::atomic qmd_tables{}; + std::atomic qmd_tables_wrong{}; + /// Of the tables, those compared right after the front valued signals late; signals valued + /// late whose descriptors were compared, those the back read otherwise, and those whose back + /// reads were gone (more than LateReadsKept waited); QMD methods the front valued after the + /// back could have run them (their entry published). + std::atomic qmd_tables_late{}; + std::atomic qmd_late_signals{}; + std::atomic qmd_late_wrong{}; + std::atomic qmd_late_missing{}; + std::atomic qmd_published_early{}; + std::mutex released_mutex; + std::unordered_set released; ///< Device addresses of the semaphores the GPU released + std::chrono::steady_clock::time_point last_report{}; ///< The front's +}; + +inline State& Get() { + static State state; + return state; +} + +/// The back runs a semaphore release at `device_address`. +inline void NoteRelease(u64 device_address) { + State& state = Get(); + std::scoped_lock lock{state.released_mutex}; + state.released.insert(device_address & ~u64{3}); +} + +[[nodiscard]] inline bool WasReleased(u64 device_address) { + State& state = Get(); + std::scoped_lock lock{state.released_mutex}; + return state.released.contains(device_address & ~u64{3}); +} + +/// An acquire, or the poll of a suspended one (`poll`), at a semaphore the GPU had `released`: +/// whether the value met it before the drain and after. +inline void NoteAcquire(bool poll, bool released, bool met_before, bool met_after) { + const size_t index = + (poll ? 8U : 0U) + (released ? 4U : 0U) + (met_before ? 2U : 0U) + (met_after ? 1U : 0U); + Get().acquires[index].fetch_add(1, std::memory_order_relaxed); +} + +/// How the semaphore view passed an acquire or resumed a poll. +enum class ViewPass : u32 { + Memory, ///< Met in memory + Queued, ///< Met by the last release sent to the word, still queued + Waited, ///< Met after a positioned wait (GpuSplit::WaitForPosition) +}; + +/// The view passed an acquire (or resumed a poll, `poll`); whether the word read after a drain +/// met it too, and if not, whether a fenced release that landed during the drain changed it +/// (`raced`). +inline void NoteViewPass(bool poll, ViewPass how, bool met_after_drain, bool raced) { + const size_t index = (poll ? 3U : 0U) + static_cast(how); + State& state = Get(); + state.view_passes[index].fetch_add(1, std::memory_order_relaxed); + if (!met_after_drain) { + (raced ? state.view_raced : state.view_wrong)[index].fetch_add(1, + std::memory_order_relaxed); + } +} + +/// The view suspended an acquire (or kept a poll waiting, `poll`); whether the word read after +/// a drain met it. +inline void NoteViewWait(bool poll, bool met_after_drain) { + State& state = Get(); + state.view_waits[poll ? 1 : 0].fetch_add(1, std::memory_order_relaxed); + if (met_after_drain) { + state.view_waits_met[poll ? 1 : 0].fetch_add(1, std::memory_order_relaxed); + } +} + +/// A release of a QMD launch: the word and what it holds once the back ran it, or a value the +/// front could not tell (`known` false), or that value at least (`at_least`). +struct QmdRelease { + u64 device; + u32 value; + bool known; + bool at_least; + + bool operator==(const QmdRelease&) const = default; +}; + +/// A QMD the scheduling read (NativeQmdScheduler::Signal's read): its GPU address and words. +struct QmdRead { + u64 address; + std::array words; +}; + +/// The releases the back ran for QMD launches since the last check (the back's thread only). +inline std::vector& BackQmdReleases() { + static std::vector released; + return released; +} + +/// The QMDs the back's scheduling read since the last check (the back's thread only). +inline std::vector& BackQmdReads() { + static std::vector reads; + return reads; +} + +/// Where one signal's reads end in BackQmdReads (no QMD lives at that address). +inline constexpr u64 SignalEnd = ~u64{0}; + +/// The descriptors the back read for the signals that waited on the front, by the position after +/// their entry and their place in it, until the front valued them late (the back's thread only). +inline std::map, std::vector>& LateBackReads() { + static std::map, std::vector> reads; + return reads; +} +inline constexpr size_t LateReadsKept = 16384; + +/// A signal the front valued late (it waited), at the entry ending at `position`, `index`-th of +/// the entry's signals, and the descriptors it read for it. +struct LateQmdRead { + u64 position; + u32 index; + std::vector reads; +}; + +/// The split's back's scheduling ran one signal (KeplerCompute::ProcessNativeMethod). +inline void NoteQmdSignalEnd(bool on_back) { + if (on_back) { + BackQmdReads().push_back({SignalEnd, {}}); + } +} + +/// The front valued a QMD method after its entry was published: the back could have run it. +inline void NotePublishedEarly() { + State& state = Get(); + state.qmd_published_early.fetch_add(1, std::memory_order_relaxed); + if (state.qmd_reads_logged.fetch_add(1, std::memory_order_relaxed) < 16) { + LOG_ERROR(HW_GPU, "QMD view check: the front valued a QMD method the back could run"); + } +} + +/// The split's back runs a release of a QMD launch (KeplerCompute::ProcessNativeMethod). +inline void NoteQmdRelease(u64 device_address, u32 value, bool on_back) { + if (on_back) { + BackQmdReleases().push_back({device_address, value, true, false}); + } +} + +/// The split's back's scheduling read a QMD (KeplerCompute::ProcessNativeMethod). +inline void NoteQmdRead(u64 address, const std::array& words, bool on_back) { + if (on_back) { + BackQmdReads().push_back({address, words}); + } +} + +/// The back compared the front's QMD table with its own at the same place (`same`); `late`: right +/// after the front valued signals late. +inline void NoteQmdTables(bool same, size_t states, bool late = false) { + State& state = Get(); + state.qmd_tables.fetch_add(1, std::memory_order_relaxed); + if (late) { + state.qmd_tables_late.fetch_add(1, std::memory_order_relaxed); + } + if (!same) { + state.qmd_tables_wrong.fetch_add(1, std::memory_order_relaxed); + if (state.qmd_reads_logged.fetch_add(1, std::memory_order_relaxed) < 16) { + LOG_ERROR(HW_GPU, + "QMD view check: the front's table ({} QMDs) differs from the back's{}", + states, late ? " after it valued signals late" : ""); + } + } +} + +/// The back, after the signals the front valued late: the descriptors it read for them against +/// the ones it read itself running them. +inline void CheckLateQmdReads(std::span late) { + State& state = Get(); + auto& kept = LateBackReads(); + for (const LateQmdRead& signal : late) { + const auto it = kept.find({signal.position, signal.index}); + if (it == kept.end()) { + state.qmd_late_missing.fetch_add(1, std::memory_order_relaxed); + continue; + } + state.qmd_late_signals.fetch_add(1, std::memory_order_relaxed); + const auto& back_reads = it->second; + bool same = back_reads.size() == signal.reads.size(); + for (size_t i = 0; same && i < back_reads.size(); ++i) { + same = back_reads[i].address == signal.reads[i].address && + back_reads[i].words == signal.reads[i].words; + } + if (!same) { + state.qmd_late_wrong.fetch_add(1, std::memory_order_relaxed); + if (state.qmd_reads_logged.fetch_add(1, std::memory_order_relaxed) < 16) { + std::string text; + for (size_t i = 0; i < (std::max)(back_reads.size(), signal.reads.size()); ++i) { + const QmdRead* back = i < back_reads.size() ? &back_reads[i] : nullptr; + const QmdRead* front = i < signal.reads.size() ? &signal.reads[i] : nullptr; + text += fmt::format(" | {:#x} by the back, {:#x} by the front", + back ? back->address : 0, front ? front->address : 0); + for (size_t word = 0; back && front && word < 64; ++word) { + if (back->words[word] != front->words[word]) { + text += fmt::format(", word {} {:#x} for {:#x}", word, + back->words[word], front->words[word]); + break; + } + } + } + LOG_ERROR(HW_GPU, + "QMD view check: the back read other descriptors for a signal the " + "front valued late (position {:#x}, signal {}){}", + signal.position, signal.index, text); + } + } + kept.erase(it); + } +} + +/// The back, right after QMD methods the front sent: what it read and released for them against +/// what the front read and noted (`front_reads`, `predicted`, in order), or nothing to compare +/// where the front could not value them (`valued` false). +/// `wait_position`: the signals of the entry ending there waited on the front from the +/// `first_waiting`-th on (0: none): their reads are kept for CheckLateQmdReads. +inline void CheckQmdReleases(std::span predicted, + std::span front_reads, bool valued, + u64 wait_position = 0, u32 first_waiting = 0) { + State& state = Get(); + auto& released = BackQmdReleases(); + auto& all_reads = BackQmdReads(); + // By signal: each ends with a SignalEnd. + std::vector back_reads; + std::vector> signals(1); + size_t ends = 0; + for (QmdRead& read : all_reads) { + if (read.address == SignalEnd) { + signals.emplace_back(); + ++ends; + continue; + } + back_reads.push_back(read); + signals.back().push_back(std::move(read)); + } + signals.resize(ends); + all_reads.clear(); + if (!valued) { + state.qmd_unvalued.fetch_add(1, std::memory_order_relaxed); + released.clear(); + auto& kept = LateBackReads(); + if (wait_position != 0) { + // Kept from before the split last started over at position 0. + kept.erase(kept.lower_bound({wait_position, 0}), kept.end()); + } + for (u32 index = first_waiting; wait_position != 0 && index < signals.size(); ++index) { + if (kept.size() >= LateReadsKept) { + kept.erase(kept.begin()); // The oldest: the front lost or dropped it + } + kept.insert_or_assign({wait_position, index}, std::move(signals[index])); + } + return; + } + state.qmd_reads.fetch_add(back_reads.size(), std::memory_order_relaxed); + bool same_reads = back_reads.size() == front_reads.size(); + for (size_t i = 0; same_reads && i < back_reads.size(); ++i) { + same_reads = back_reads[i].address == front_reads[i].address && + back_reads[i].words == front_reads[i].words; + } + if (!same_reads) { + state.qmd_reads_wrong.fetch_add(1, std::memory_order_relaxed); + if (state.qmd_reads_logged.fetch_add(1, std::memory_order_relaxed) < 16) { + std::string text; + for (size_t i = 0; i < (std::max)(back_reads.size(), front_reads.size()); ++i) { + const QmdRead* back = i < back_reads.size() ? &back_reads[i] : nullptr; + const QmdRead* front = i < front_reads.size() ? &front_reads[i] : nullptr; + text += fmt::format(" | {:#x} by the back, {:#x} by the front", + back ? back->address : 0, front ? front->address : 0); + for (size_t word = 0; back && front && word < 64; ++word) { + if (back->words[word] != front->words[word]) { + text += fmt::format(", word {} {:#x} for {:#x}", word, back->words[word], + front->words[word]); + break; + } + } + } + LOG_ERROR(HW_GPU, "QMD view check: the back read other descriptors{}", text); + } + } + state.qmd_methods.fetch_add(1, std::memory_order_relaxed); + state.qmd_releases.fetch_add(released.size(), std::memory_order_relaxed); + bool same = released.size() == predicted.size(); + for (size_t i = 0; same && i < released.size(); ++i) { + const QmdRelease& note = predicted[i]; + same = released[i].device == note.device && + (!note.known || (note.at_least ? released[i].value >= note.value + : released[i].value == note.value)); + } + if (!same) { + state.qmd_wrong.fetch_add(1, std::memory_order_relaxed); + if (state.qmd_logged.fetch_add(1, std::memory_order_relaxed) < 16) { + std::string text; + for (const QmdRelease& release : released) { + text += fmt::format(" {:#x}={:#x}", release.device, release.value); + } + text += " | noted"; + for (const QmdRelease& note : predicted) { + text += fmt::format(" {:#x}{}{:#x}", note.device, + !note.known ? "=?" + : note.at_least ? ">=" + : "=", + note.value); + } + LOG_ERROR(HW_GPU, "QMD view check: the back released{}", text); + } + } + released.clear(); +} + +/// The front fetches a list without waiting for writers still queued after the last sync. +inline void NoteListUnwaited() { + Get().lists_unwaited.fetch_add(1, std::memory_order_relaxed); +} + +/// The back reached that list: whether the work before it made the list's pages GPU-written. +inline void NoteListChecked(bool written) { + if (written) { + Get().lists_written.fetch_add(1, std::memory_order_relaxed); + } +} + +/// The front, between commands. +inline void MaybeReport() { + if (!g_enabled) { + return; + } + State& state = Get(); + const auto now = std::chrono::steady_clock::now(); + if (state.last_report == std::chrono::steady_clock::time_point{}) { + state.last_report = now; + return; + } + const double seconds = std::chrono::duration(now - state.last_report).count(); + if (seconds < 10.0) { + return; + } + state.last_report = now; + const auto take = [](std::atomic& value) { + return value.exchange(0, std::memory_order_relaxed); + }; + const auto group = [&](bool poll, bool released) { + const size_t base = (poll ? 8U : 0U) + (released ? 4U : 0U); + const u64 neither = take(state.acquires[base]); + const u64 only_after = take(state.acquires[base + 1]); + const u64 only_before = take(state.acquires[base + 2]); + const u64 both = take(state.acquires[base + 3]); + return fmt::format("{} (met before and after the drain {}, only after {}, only before {}, " + "neither {})", + neither + only_after + only_before + both, both, only_after, only_before, + neither); + }; + const auto passes = [&](bool poll) { + const size_t base = poll ? 3U : 0U; + const u64 by_memory = take(state.view_passes[base]); + const u64 by_queued = take(state.view_passes[base + 1]); + const u64 waited = take(state.view_passes[base + 2]); + const u64 wrong_memory = take(state.view_wrong[base]); + const u64 wrong_queued = take(state.view_wrong[base + 1]); + const u64 wrong_waited = take(state.view_wrong[base + 2]); + const u64 raced = take(state.view_raced[base]) + take(state.view_raced[base + 1]) + + take(state.view_raced[base + 2]); + return fmt::format("{} (by memory {}, by a release still queued {}, after a positioned " + "wait {}), not met after the drain {} (of those by memory {}, queued " + "{}, after a wait {}), changed by a fenced release landing during the " + "drain {}", + by_memory + by_queued + waited, by_memory, by_queued, waited, + wrong_memory + wrong_queued + wrong_waited, wrong_memory, wrong_queued, + wrong_waited, raced); + }; + const auto waits = [&](bool poll) { + const u64 count = take(state.view_waits[poll ? 1 : 0]); + const u64 met = take(state.view_waits_met[poll ? 1 : 0]); + return fmt::format("{} (met after the drain {})", count, met); + }; + const u64 unwaited = take(state.lists_unwaited); + const u64 written = take(state.lists_written); + const std::string acquires_never = group(false, false); + const std::string acquires_released = group(false, true); + const std::string polls_never = group(true, false); + const std::string polls_released = group(true, true); + const std::string acquire_passes = passes(false); + const std::string acquire_waits = waits(false); + const std::string poll_passes = passes(true); + const std::string poll_waits = waits(true); + const u64 qmd_methods = take(state.qmd_methods); + const u64 qmd_releases = take(state.qmd_releases); + const u64 qmd_wrong = take(state.qmd_wrong); + const u64 qmd_unvalued = take(state.qmd_unvalued); + const u64 qmd_reads = take(state.qmd_reads); + const u64 qmd_reads_wrong = take(state.qmd_reads_wrong); + const u64 qmd_tables = take(state.qmd_tables); + const u64 qmd_tables_wrong = take(state.qmd_tables_wrong); + const u64 qmd_tables_late = take(state.qmd_tables_late); + const u64 qmd_late_signals = take(state.qmd_late_signals); + const u64 qmd_late_wrong = take(state.qmd_late_wrong); + const u64 qmd_late_missing = take(state.qmd_late_missing); + const u64 qmd_published_early = take(state.qmd_published_early); + LOG_INFO(HW_GPU, + "Drain checks in {:.1f} s: command lists that did not wait for writers sent after the " + "last sync {}, GPU-written by those writers {} | drained acquires at addresses the " + "GPU never released {}, at ones it released {} | drained polls of suspended acquires " + "at addresses never released {}, at released ones {} | semaphore view: acquires " + "passed {}, suspended {}; polls resumed {}, kept waiting {} | QMD view: methods " + "valued {} with {} releases, released otherwise than noted {}, with {} descriptors " + "read, read otherwise {}; valued after the back could run them {}; signals valued " + "late {}, read otherwise {}, not compared {}; tables compared {} (after valuing late " + "{}), differing {}; not valued {}", + seconds, unwaited, written, acquires_never, acquires_released, polls_never, + polls_released, acquire_passes, acquire_waits, poll_passes, poll_waits, qmd_methods, + qmd_releases, qmd_wrong, qmd_reads, qmd_reads_wrong, qmd_published_early, + qmd_late_signals, qmd_late_wrong, qmd_late_missing, qmd_tables, qmd_tables_late, + qmd_tables_wrong, qmd_unvalued); +} + +} // namespace VideoCommon::Nx2DrainCheck + +// EDEN_NX2_ACQUIRE_STATS: how the split's front decides Switch 2 semaphore acquires (Puller), every +// 10 s, without changing what runs: by operation, which of them still wait, at which words (on +// GPU-written pages, by the releases the front sent there), what resumes them and how long they +// waited. Front thread only, but for the writer that latched a waiter (Nx2SemaphoreWaiters). +namespace VideoCommon::Nx2AcquireStats { + +// A plain flag: the front checks it at every acquire and release it sends. +inline bool g_enabled = std::getenv("EDEN_NX2_ACQUIRE_STATS") != nullptr; + +/// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads +/// nx2_switches.txt, so a switch set only there takes effect from that point on. +inline void ReadSwitch() { + g_enabled = std::getenv("EDEN_NX2_ACQUIRE_STATS") != nullptr; +} + +/// How the front decided an acquire. +enum class Decision : u32 { + Memory, ///< Passed: met in memory + Queued, ///< Passed: met by the last release sent to the word, still queued + Waited, ///< Passed after a positioned wait + Suspended, ///< The channel waits + Drained, ///< The semaphore view doesn't apply: drained as before + Count, +}; + +/// Who the front sent releases to a word for (State::noted bits). +enum class Releaser : u32 { + Puller, ///< Semaphore methods (WriteLong, SEM_EXECUTE release) + Report3D, ///< 3D report semaphores + ReportCompute, ///< Compute report semaphores + Copy, ///< Copy engine semaphores + Qmd, ///< QMD launches the front valued (FrontQmdView) + Count, +}; + +/// Why the front's QMD view could not value a signal (FrontQmdView::Signal), or one that waited. +enum class QmdLost : u32 { + NotSynced, ///< Its table is not the back's yet (the back has not caught up since) + Unplaced, ///< Work that may write guest memory anywhere was still queued + GpuWritten, ///< A QMD it read is on a page GPU work wrote + Pending, ///< A write into a QMD it read was still queued + ReleasedInto, ///< A release of the same signal writes into a QMD it read + Mappings, ///< Work that may change the GPU's mappings was still queued + WrittenAfter, ///< Waiting, a QMD it reads was written after it + MappedAfter, ///< Waiting, the mappings may have changed after it + LateReleasedInto, ///< Waiting, a release of it writes a QMD read late + TooMany, ///< Too many signals waited + Count, +}; + +/// Where the front's QMD view took the back's table over (FrontQmdView::TakeOver). +enum class QmdTakeOver : u32 { + FrontStart, ///< An engine's front started, with the back drained + BetweenCommands, ///< The back had run everything sent (GPU thread loop) + QmdMethod, ///< The same, at a QMD method + Count, +}; + +/// What the QMD view noted for a release. +enum class QmdNote : u32 { + Exact, + AtLeast, ///< A maximum with a word it can't tell + Unknown, ///< A reduction of a word it can't tell + Count, +}; + +/// What resumed a suspended acquire. +enum class Resume : u32 { + Latch, ///< A release the back ran latched the waiter (by its Source) + Memory, ///< Met in memory: the CPU, or a GPU write that latches no waiter + Queued, ///< Met by the last release sent to the word, still queued + Drain, ///< EDEN_NX2_DRAIN_CHECK's drain met it first + Count, +}; + +using Source = Tegra::Nx2SemaphoreWaiters::Source; + +struct State { + /// By operation (equal, greater or equal, mask) and Decision. + std::array(Decision::Count)>, 3> acquires{}; + u64 positioned_waits{}; + u64 suspended_gpu_written{}; ///< Words on GPU-written pages + u64 suspended_flush{}; ///< Of those, words the buffer cache held newer than memory + std::array(Releaser::Count)> suspended_noted{}; + u64 suspended_never_noted{}; ///< No release the front sent: QMD releases, the CPU + std::array(Resume::Count)> resumes{}; + std::array(Source::Count)> latches{}; + u64 resumes_after_flush{}; + u64 wait_ns{}; + u64 wait_max_ns{}; + u64 waits_over_1ms{}; + u64 passed_by_qmd{}; ///< Of the passes by a queued release, at words QMD launches release + u64 qmd_valued{}; + std::array(QmdLost::Count)> qmd_lost{}; + /// Of qmd_lost[Unplaced] and qmd_lost[Mappings]: by what sent the last such write + std::array(UnplacedKind::Count)> qmd_lost_unplaced{}; + std::array(UnplacedKind::Count)> qmd_lost_mappings{}; + std::array(QmdNote::Count)> qmd_notes{}; + u64 qmd_inline_reads{}; ///< Descriptors read from an inline QMD still queued + /// Signals that waited first, by why, and behind one; valued late, and how often the last + /// waiting one was; waits that lost the table, by why, and the waiting signals they dropped; + /// waiting signals takeovers dropped (the back had run them). + std::array(QmdLost::Count)> qmd_waits{}; + u64 qmd_behind{}; + u64 qmd_late{}; + u64 qmd_caught_up{}; + std::array(QmdLost::Count)> qmd_late_lost{}; + u64 qmd_late_dropped{}; + u64 qmd_takeover_dropped{}; + /// Of qmd_waits[Pending]: by what wrote the QMD (FrontQmdView::NoteWaitWriter) + std::map qmd_wait_writers; + std::array(QmdTakeOver::Count)> qmd_syncs{}; + std::chrono::steady_clock::duration qmd_sync_time{}; + std::unordered_map noted; ///< Releaser bits by word + std::chrono::steady_clock::time_point last_report{}; +}; + +inline State& Get() { + static State state; + return state; +} + +[[nodiscard]] inline size_t OperationIndex(u32 operation) { + switch (operation) { + case Tegra::Nx2SemaphoreWaiters::AcquireEqual: + return 0; + case Tegra::Nx2SemaphoreWaiters::AcquireGequal: + return 1; + default: + return 2; + } +} + +/// The front sent a release to the word at `device_address`. +inline void NoteRelease(u64 device_address, Releaser releaser) { + if (!g_enabled) { + return; + } + State& state = Get(); + if (state.noted.size() < 65536) { + state.noted[device_address & ~u64{3}] |= static_cast(1U << static_cast(releaser)); + } +} + +/// The QMD view valued a signal (`lost` empty) or not; `kind`: what sent the last write that +/// may write anywhere (Unplaced) or change the mappings (Mappings). +inline void NoteQmdSignal(std::optional lost, UnplacedKind kind = UnplacedKind::Other) { + if (!g_enabled) { + return; + } + State& state = Get(); + if (!lost) { + ++state.qmd_valued; + return; + } + ++state.qmd_lost[static_cast(*lost)]; + if (*lost == QmdLost::Unplaced) { + ++state.qmd_lost_unplaced[static_cast(kind)]; + } else if (*lost == QmdLost::Mappings) { + ++state.qmd_lost_mappings[static_cast(kind)]; + } +} + +/// A signal the QMD view could not value at once waits: the first by `why` (`kind`: what sent the +/// last write that may write anywhere or change the mappings), or behind one (`why` empty). +inline void NoteQmdWait(std::optional why, UnplacedKind kind = UnplacedKind::Other) { + if (!g_enabled) { + return; + } + State& state = Get(); + if (!why) { + ++state.qmd_behind; + return; + } + ++state.qmd_waits[static_cast(*why)]; + if (*why == QmdLost::Unplaced) { + ++state.qmd_lost_unplaced[static_cast(kind)]; + } else if (*why == QmdLost::Mappings) { + ++state.qmd_lost_mappings[static_cast(kind)]; + } +} + +/// A signal waits for a write into a QMD by `writer`, still queued. +inline void NoteQmdWaitWriter(const std::string& writer) { + if (g_enabled) { + ++Get().qmd_wait_writers[writer]; + } +} + +/// The QMD view valued a waiting signal late; `caught_up`: none waits now. +inline void NoteQmdLate(bool caught_up) { + if (g_enabled) { + State& state = Get(); + ++state.qmd_late; + state.qmd_caught_up += caught_up ? 1 : 0; + } +} + +/// The QMD view lost its table valuing a waiting signal late (`why`), dropping `dropped` waiting +/// signals. +inline void NoteQmdLateLost(QmdLost why, size_t dropped) { + if (g_enabled) { + State& state = Get(); + ++state.qmd_late_lost[static_cast(why)]; + state.qmd_late_dropped += dropped; + } +} + +/// A takeover dropped `dropped` waiting signals: the back had run them. +inline void NoteQmdTakeOverDropped(size_t dropped) { + if (g_enabled) { + Get().qmd_takeover_dropped += dropped; + } +} + +/// The QMD view read a descriptor from an inline QMD still queued. +inline void NoteQmdInlineRead() { + if (g_enabled) { + ++Get().qmd_inline_reads; + } +} + +/// The QMD view noted a release so. +inline void NoteQmdNote(QmdNote note) { + if (g_enabled) { + ++Get().qmd_notes[static_cast(note)]; + } +} + +/// The QMD view took the back's table over, at `where`, in `time`. +inline void NoteQmdSync(QmdTakeOver where, std::chrono::steady_clock::duration time) { + if (g_enabled) { + State& state = Get(); + ++state.qmd_syncs[static_cast(where)]; + state.qmd_sync_time += time; + } +} + +/// An acquire of `operation` (Puller GpuSemaphoreOperation) at `device_address`, decided so. +/// Suspended: whether its word is on a GPU-written page, and the buffer cache held it newer. +inline void NoteAcquire(u32 operation, Decision decision, u64 device_address, bool waited, + bool gpu_written, bool flush) { + State& state = Get(); + ++state.acquires[OperationIndex(operation)][static_cast(decision)]; + if (waited) { + ++state.positioned_waits; + } + if (decision == Decision::Queued) { + const auto it = state.noted.find(device_address & ~u64{3}); + if (it != state.noted.end() && ((it->second >> static_cast(Releaser::Qmd)) & 1)) { + ++state.passed_by_qmd; + } + } + if (decision != Decision::Suspended) { + return; + } + if (gpu_written) { + ++state.suspended_gpu_written; + } + if (flush) { + ++state.suspended_flush; + } + const auto it = state.noted.find(device_address & ~u64{3}); + if (it == state.noted.end()) { + ++state.suspended_never_noted; + return; + } + for (u32 releaser = 0; releaser < static_cast(Releaser::Count); ++releaser) { + if ((it->second >> releaser) & 1) { + ++state.suspended_noted[releaser]; + } + } +} + +/// A suspended acquire resumed, after `waited`. +inline void NoteResume(Resume how, Source latch_source, bool after_flush, + std::chrono::steady_clock::duration waited) { + State& state = Get(); + ++state.resumes[static_cast(how)]; + if (how == Resume::Latch) { + ++state.latches[static_cast(latch_source)]; + } + if (after_flush) { + ++state.resumes_after_flush; + } + const u64 ns = static_cast(std::max( + 0, std::chrono::duration_cast(waited).count())); + state.wait_ns += ns; + state.wait_max_ns = std::max(state.wait_max_ns, ns); + if (ns > 1'000'000) { + ++state.waits_over_1ms; + } +} + +/// The front, between commands. +inline void MaybeReport() { + if (!g_enabled) { + return; + } + State& state = Get(); + const auto now = std::chrono::steady_clock::now(); + if (state.last_report == std::chrono::steady_clock::time_point{}) { + state.last_report = now; + return; + } + const double seconds = std::chrono::duration(now - state.last_report).count(); + if (seconds < 10.0) { + return; + } + state.last_report = now; + const auto by_operation = [&](size_t operation) { + const auto& counts = state.acquires[operation]; + u64 total = 0; + for (const u64 count : counts) { + total += count; + } + return fmt::format("{} (passed by memory {}, by a release still queued {}, after a " + "positioned wait {}; suspended {}; drained {})", + total, counts[0], counts[1], counts[2], counts[3], counts[4]); + }; + const auto& noted = state.suspended_noted; + const auto& resumes = state.resumes; + const auto& latches = state.latches; + const u64 resumed = resumes[0] + resumes[1] + resumes[2] + resumes[3]; + LOG_INFO(HW_GPU, + "Acquires in {:.1f} s (EDEN_NX2_ACQUIRE_STATS): equal {}, greater or equal {}, mask " + "{}; positioned waits {}; passed by a queued QMD launch's release {} | suspended at " + "words on GPU-written pages {} (held newer by the buffer cache {}), with releases " + "sent by the puller {}, 3D reports {}, compute reports {}, the copy engine {}, QMD " + "launches the front valued {}, none (other QMD releases, the CPU) {} | resumed {}: " + "latched {} (payload releases {}, query results {}, QMD reductions {}, QMD maxima {}, " + "other {}), by memory {}, by a queued release {}, by the drain check {}, after a " + "flush {} | waited {:.1f} ms, at most {:.3f} ms, over 1 ms {}", + seconds, by_operation(0), by_operation(1), by_operation(2), state.positioned_waits, + state.passed_by_qmd, state.suspended_gpu_written, state.suspended_flush, noted[0], + noted[1], noted[2], noted[3], noted[4], state.suspended_never_noted, resumed, + resumes[0], latches[1], latches[2], latches[3], latches[4], latches[0], resumes[1], + resumes[2], resumes[3], state.resumes_after_flush, + static_cast(state.wait_ns) / 1e6, static_cast(state.wait_max_ns) / 1e6, + state.waits_over_1ms); + const auto& lost = state.qmd_lost; + const auto& notes = state.qmd_notes; + const auto& syncs = state.qmd_syncs; + const auto kinds = [](const auto& counts) { + std::string text; + for (size_t kind = 0; kind < counts.size(); ++kind) { + if (counts[kind] != 0) { + text += fmt::format("{}{} {}", text.empty() ? " (" : ", ", + unplaced_kind_names[kind], counts[kind]); + } + } + return text.empty() ? text : text + ")"; + }; + // The writers a signal waited for most. + std::vector> writers; + for (const auto& [writer, count] : state.qmd_wait_writers) { + writers.emplace_back(count, writer); + } + std::sort(writers.rbegin(), writers.rend()); + std::string by_writer; + for (size_t i = 0; i < writers.size() && i < 6; ++i) { + by_writer += + fmt::format("{}{} {}", i == 0 ? " (by " : ", ", writers[i].second, writers[i].first); + } + if (!by_writer.empty()) { + by_writer += ")"; + } + const auto& waits = state.qmd_waits; + const auto& late_lost = state.qmd_late_lost; + const auto index = [](QmdLost why) { return static_cast(why); }; + u64 waited_first = 0; + u64 lost_late = 0; + for (size_t why = 0; why < waits.size(); ++why) { + waited_first += waits[why]; + lost_late += late_lost[why]; + } + LOG_INFO( + HW_GPU, + "QMD view in {:.1f} s (EDEN_NX2_ACQUIRE_STATS): signals valued {}, valued late {} " + "(caught up {} times) | waited first {} (a write into the QMD queued {}{}, work " + "that may write anywhere queued {}{}, a change of the mappings queued {}{}), behind an " + "earlier one {}; lost while waiting {} (a QMD written after it {}, the mappings " + "changed after it {}, a release into a QMD read late {}, too many waiting {}; {} " + "waiting dropped), dropped at takeovers {} | not valued: table not synced yet {}, " + "QMD on a GPU-written page {}, a release of the signal into it {} | descriptors read " + "from an inline QMD still queued {} | releases noted exact {}, at least {}, unknown " + "{} | tables taken over {} in {:.3f} ms (at a front's start {}, between commands {}, " + "at QMD methods {})", + seconds, state.qmd_valued, state.qmd_late, state.qmd_caught_up, waited_first, + waits[index(QmdLost::Pending)], by_writer, waits[index(QmdLost::Unplaced)], + kinds(state.qmd_lost_unplaced), waits[index(QmdLost::Mappings)], + kinds(state.qmd_lost_mappings), state.qmd_behind, lost_late, + late_lost[index(QmdLost::WrittenAfter)], late_lost[index(QmdLost::MappedAfter)], + late_lost[index(QmdLost::LateReleasedInto)], late_lost[index(QmdLost::TooMany)], + state.qmd_late_dropped, state.qmd_takeover_dropped, lost[index(QmdLost::NotSynced)], + lost[index(QmdLost::GpuWritten)], lost[index(QmdLost::ReleasedInto)], + state.qmd_inline_reads, notes[0], notes[1], notes[2], syncs[0] + syncs[1] + syncs[2], + std::chrono::duration(state.qmd_sync_time).count(), syncs[0], syncs[1], + syncs[2]); + auto noted_words = std::move(state.noted); + state = State{}; + state.noted = std::move(noted_words); + state.last_report = now; +} + +} // namespace VideoCommon::Nx2AcquireStats diff --git a/src/video_core/nx2_entry_check.h b/src/video_core/nx2_entry_check.h new file mode 100644 index 0000000000..29004b2585 --- /dev/null +++ b/src/video_core/nx2_entry_check.h @@ -0,0 +1,244 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "common/common_types.h" +#include "common/logging.h" +#include "video_core/cache_types.h" +#include "video_core/dma_pusher.h" +#include "video_core/gpu_split.h" +#include "video_core/memory_manager.h" +#include "video_core/nx2.h" +#include "video_core/nx2_drain_check.h" +#include "video_core/rasterizer_interface.h" + +// TEST: checks of the command entries DmaPusher::Step parses without the GPU-written path, from +// their submit-time copy or from guest memory. +// +// EDEN_NX2_DRAIN_CHECK: the entries that began between methods and so did not wait for the +// compute work sent before the last sync while some was queued. At each one's place the back +// looks for pages that work has made GPU-written by then, brings the GPU's bytes of those to guest +// memory and compares the words the front parsed: differing ones must be 0. +// +// EDEN_NX2_ENTRY_CHECK: every such entry compared at its place on the back with guest memory: +// words written after the front took them, by the CPU or by GPU work that writes guest memory +// past the caches (inline uploads, semaphore releases, reports), which no page bit tells about. +// +// One line every 10 s from the GPU thread. +namespace Tegra::Nx2EntryCheck { + +// A plain flag: checked at every command entry. +inline bool g_entries = std::getenv("EDEN_NX2_ENTRY_CHECK") != nullptr; + +/// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads +/// nx2_switches.txt, so a switch set only there takes effect from that point on. +inline void ReadSwitch() { + g_entries = std::getenv("EDEN_NX2_ENTRY_CHECK") != nullptr; +} + +[[nodiscard]] inline bool Any() { + return g_entries || VideoCommon::Nx2DrainCheck::g_enabled; +} + +/// What the front parsed: the first words and a hash of all. +struct Parsed { + std::array head{}; + u32 words{}; + u64 hash{}; +}; + +[[nodiscard]] inline Parsed Summarize(std::span words) { + Parsed parsed; + parsed.words = static_cast(words.size()); + parsed.hash = HashPushbufferWords(words); + for (size_t i = 0; i < parsed.head.size() && i < words.size(); ++i) { + parsed.head[i] = words[i]; + } + return parsed; +} + +/// The words of guest memory at [address, address + words * 4). +[[nodiscard]] inline Parsed SummarizeMemory(MemoryManager& memory, GPUVAddr address, u32 words) { + std::vector data(words); + if (words != 0) { + memory.ReadBlockUnsafe(address, data.data(), data.size() * sizeof(u32)); + } + return Summarize(data); +} + +enum Path : u32 { Copy, Memory, PathCount }; + +struct State { + // Front. + std::atomic skipped{}; ///< Entries that did not wait for queued writers + std::atomic kept_for_copies{}; ///< Command entries that waited for queued copy work + std::array, PathCount> compared{}; + // Back. + std::atomic skipped_checked{}; + std::atomic skipped_written{}; ///< On a page GPU work had made GPU-written at its place + std::atomic skipped_differ{}; ///< Of those, words differing after a flush + std::array, PathCount> differ{}; + std::atomic logged{}; + std::chrono::steady_clock::time_point last_log{std::chrono::steady_clock::now()}; +}; + +inline State& Get() { + static State state; + return state; +} + +[[nodiscard]] inline std::string Hex(std::span words, u32 count) { + std::string text; + for (u32 i = 0; i < count && i < words.size(); ++i) { + text += fmt::format("{}{:08x}", text.empty() ? "" : " ", words[i]); + } + return text.empty() ? std::string{"-"} : text; +} + +inline void LogDiffer(const char* what, GPUVAddr address, const Parsed& parsed, + const Parsed& memory) { + State& state = Get(); + if (state.logged.fetch_add(1, std::memory_order_relaxed) >= 16) { + return; + } + LOG_WARNING(HW_GPU, + "Command entry check: {} at {:#x} ({} words): parsed starts {}, guest memory " + "{} at the back's place", + what, address, parsed.words, Hex(parsed.head, parsed.words), + Hex(memory.head, memory.words)); +} + +/// Back, at the place of an entry that did not wait for the writers before the last sync: were +/// its words written by that work? +inline void CheckSkipped(MemoryManager* memory, VideoCore::RasterizerInterface* rasterizer, + GPUVAddr address, Parsed parsed) { + State& state = Get(); + state.skipped_checked.fetch_add(1, std::memory_order_relaxed); + const GPUVAddr end = address + u64{parsed.words} * sizeof(u32); + bool written = false; + for (GPUVAddr page = address & ~GPUVAddr{0xfff}; page < end; page += 0x1000) { + const auto device = memory->GpuToCpuAddress(page); + written = written || (device && Nx2IsGpuWritten(*device)); + } + if (!written) { + return; + } + state.skipped_written.fetch_add(1, std::memory_order_relaxed); + if (rasterizer != nullptr) { + // The GPU's bytes of the entry's pages, in guest memory for the comparison. + for (GPUVAddr piece = address; piece < end;) { + const GPUVAddr piece_end = (std::min)(end, (piece & ~GPUVAddr{0xfff}) + 0x1000); + if (const auto device = memory->GpuToCpuAddress(piece)) { + rasterizer->FlushRegion(*device, piece_end - piece, + VideoCommon::CacheType::BufferCache); + } + piece = piece_end; + } + } + const Parsed now = SummarizeMemory(*memory, address, parsed.words); + if (now.hash != parsed.hash) { + state.skipped_differ.fetch_add(1, std::memory_order_relaxed); + LogDiffer("an entry that did not wait for the writers differs from GPU-written memory", + address, parsed, now); + } +} + +/// Back, at an entry's place (EDEN_NX2_ENTRY_CHECK): does guest memory still hold what the front +/// parsed? +inline void CheckCompared(MemoryManager* memory, GPUVAddr address, Parsed parsed, Path path) { + const Parsed now = SummarizeMemory(*memory, address, parsed.words); + if (now.hash == parsed.hash) { + return; + } + Get().differ[path].fetch_add(1, std::memory_order_relaxed); + LogDiffer(path == Copy ? "an entry run from its submit-time copy differs from guest memory" + : "an entry run from guest memory differs from guest memory later", + address, parsed, now); +} + +inline void MaybeLog() { + State& state = Get(); + const auto now = std::chrono::steady_clock::now(); + const double seconds = std::chrono::duration(now - state.last_log).count(); + if (seconds < 10.0) { + return; + } + state.last_log = now; + const auto take = [](std::atomic& counter) { + return counter.exchange(0, std::memory_order_relaxed); + }; + const u64 skipped = take(state.skipped); + const u64 kept = take(state.kept_for_copies); + const u64 checked = take(state.skipped_checked); + const u64 written = take(state.skipped_written); + const u64 differ = take(state.skipped_differ); + const u64 copies = take(state.compared[Copy]); + const u64 copies_differ = take(state.differ[Copy]); + const u64 reads = take(state.compared[Memory]); + const u64 reads_differ = take(state.differ[Memory]); + LOG_INFO(HW_GPU, + "Command entry checks in {:.1f} s: entries that did not wait for queued writers {} " + "({} checked at their place, on pages GPU work had made GPU-written {}, their words " + "differing after a flush {}), command entries that waited for queued copy work {} | " + "entries compared with guest memory at their place: run from their submit-time copy " + "{} ({} differ), from guest memory {} ({} differ)", + seconds, skipped, checked, written, differ, kept, copies, copies_differ, reads, + reads_differ); +} + +/// Front, before it parses the entry `header` of `pusher` from its submit-time copy (`copy`, else +/// guest memory). `skipped_writers`: the entry did not wait for the writers before the last sync; +/// `kept_for_copies`: a command entry that waited for them because copy work was queued. +inline void NoteEntry(DmaPusher& pusher, const CommandListHeader& header, + std::span copy, bool skipped_writers, + bool kept_for_copies) { + MaybeLog(); + auto& split = VideoCommon::GpuSplit::Get(); + if (!split.Active()) { + return; + } + const GPUVAddr address = header.addr.Value(); + const auto words = static_cast(header.size.Value()); + Parsed parsed; + if (!copy.empty()) { + std::vector data(copy.size()); + for (size_t i = 0; i < copy.size(); ++i) { + data[i] = copy[i].argument; + } + parsed = Summarize(data); + } else { + parsed = SummarizeMemory(pusher.memory_manager, address, words); + } + State& state = Get(); + if (VideoCommon::Nx2DrainCheck::g_enabled) { + if (kept_for_copies && split.WritersPending()) { + state.kept_for_copies.fetch_add(1, std::memory_order_relaxed); + } + if (skipped_writers && split.WritersPending()) { + state.skipped.fetch_add(1, std::memory_order_relaxed); + split.Defer([memory = &pusher.memory_manager, rasterizer = pusher.rasterizer, address, + parsed] { CheckSkipped(memory, rasterizer, address, parsed); }, + VideoCommon::GpuSplit::Writes::Caches); + } + } + if (g_entries) { + const Path path = copy.empty() ? Memory : Copy; + state.compared[path].fetch_add(1, std::memory_order_relaxed); + split.Defer([memory = &pusher.memory_manager, address, parsed, + path] { CheckCompared(memory, address, parsed, path); }, + VideoCommon::GpuSplit::Writes::Noted); + } +} + +} // namespace Tegra::Nx2EntryCheck diff --git a/src/video_core/nx2_fast_draw.h b/src/video_core/nx2_fast_draw.h index 13cbb253f0..9e90c92930 100644 --- a/src/video_core/nx2_fast_draw.h +++ b/src/video_core/nx2_fast_draw.h @@ -17,9 +17,11 @@ class MemoryManager; /// A draw that follows a draw of the same pipeline with only light state changes in between takes /// the delta path. It keeps the previous draw's pipeline, image views, samplers and texture buffer -/// bindings, and resolves and binds every buffer (index, vertex, uniform, storage, texture) again -/// as a full resolve does, so guest memory written without a notification is streamed as in a -/// full resolve. +/// bindings. Buffers are resolved and bound again, except small streamed uniforms whose ranges +/// were not rebound or written: those can keep the previous stream allocation in the same command +/// buffer. This assumes no unsynchronized guest CPU writes unknown to the caches between draws; +/// EDEN_NX2_UBO_KEEP_VERIFY checks that assumption. Kept image descriptors and rescaling words are +/// replayed while the views remain valid; EDEN_NX2_DELTA_DESCRIPTORS_VERIFY checks the replay. /// Light changes are draw parameters, index buffer and vertex stream registers, constant buffer /// binds and inline constant buffer data (collected in g_delta: the rasterizer resolves in full /// when they reach the pipeline's texture handles at their GPU addresses or through another diff --git a/src/video_core/nx2_list_indirect.h b/src/video_core/nx2_list_indirect.h new file mode 100644 index 0000000000..784524e7f9 --- /dev/null +++ b/src/video_core/nx2_list_indirect.h @@ -0,0 +1,1118 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common/common_types.h" +#include "video_core/gpu_split.h" +#include "video_core/macro_turing.h" + +// GPU-written command lists whose words the host GPU takes over (EDEN_NX2_LIST_INDIRECT, on +// unless =0). Ocarina of Time's compute culling writes the group counts of a later dispatch as +// the parameters of a macro call (compute method 0xe74): the split GPU thread's front had to +// wait for that work to run on the host GPU and its copy to come back before it could parse +// them (~145 ms of every second). The macros only move those words: 0xe74 stores them in the +// data RAM, 0xe7a uploads them with the dispatch's constant buffer (an inline upload), 0xe76 +// writes them into the inline QMD as its grid. So the front need not know them: +// - At the list's place, the back copies the words on the host GPU into a slot of its own (the +// snapshot: the list may be written again before the QMD launches). +// - The front parses placeholders for them, values that name the slot and the word, and runs the +// macros that see them in a tagged dry run (RunTagged): a tagged value may be moved, stored in +// the data RAM or emitted as an inline upload's data or as an inline QMD's grid, nothing else +// (no branch, computation or address depends on it). Any other use, and any 3D macro that +// may read a tagged data RAM word, concretizes the words first: the front reads the slot back +// and goes on with the real words, as before. +// - The back dispatches a QMD whose grid words are one slot's placeholders indirectly from the +// slot, and patches the uploaded words into the constant buffers compute dispatches bind +// (the "owed" words), from the slot. +namespace Tegra::Nx2ListIndirect { + +[[nodiscard]] inline bool SwitchSet() { + const char* value = std::getenv("EDEN_NX2_LIST_INDIRECT"); + return value == nullptr || value[0] != '0'; +} + +// A plain flag: checked at every GPU-written list. +inline bool g_enabled = SwitchSet(); + +/// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads +/// nx2_switches.txt, so a switch set only there takes effect from that point on. +inline void ReadSwitch() { + g_enabled = SwitchSet(); +} + +/// Slots the back keeps snapshots of list words in, and the words a slot holds. +inline constexpr u32 SlotCount = 4096; +inline constexpr u32 SlotWords = 3; + +/// The value the front carries for word `word` of the list in `slot`. A real grid never has a +/// width this large (it is over 4 billion groups). Word 0 is never 0 and the low 16 bits of words +/// 1 and 2 never are, so the QMD scheduler's empty grid check never drops a dispatch whose grid +/// is placeholders (the host GPU dispatches no groups for an empty one). +inline constexpr u32 PlaceholderMagic = 0xF1A90000U; +static_assert((SlotCount << 2) <= 0x10000); + +[[nodiscard]] constexpr u32 Placeholder(u32 slot, u32 word) { + return PlaceholderMagic | (slot << 2) | word; +} +[[nodiscard]] constexpr bool IsPlaceholder(u32 value) { + return (value & 0xFFFF0000U) == PlaceholderMagic && (value & 3) < SlotWords; +} +[[nodiscard]] constexpr u32 PlaceholderSlot(u32 value) { + return (value & 0xFFFFU) >> 2; +} +[[nodiscard]] constexpr u32 PlaceholderWord(u32 value) { + return value & 3; +} +static_assert(Placeholder(0, 0) != 0 && (Placeholder(0, 1) & 0xFFFF) != 0 && + (Placeholder(0, 2) & 0xFFFF) != 0); + +/// The slot whose three words a QMD's grid (words 12-14) is, if it is one. +[[nodiscard]] constexpr std::optional GridSlot(u32 x, u32 y, u32 z) { + if (!IsPlaceholder(x) || x != Placeholder(PlaceholderSlot(x), 0)) { + return std::nullopt; + } + const u32 slot = PlaceholderSlot(x); + if (y != Placeholder(slot, 1) || z != Placeholder(slot, 2)) { + return std::nullopt; + } + return slot; +} + +/// What a tagged dry run wrote, in order: emits (method, value) and data RAM stores (word, +/// value), `tagged` where the value is a list word moved there unchanged. +struct TaggedWrite { + u32 address; + u32 value; + bool tagged; +}; + +struct TaggedRun { + enum class Outcome { + Done, ///< Ran to its end: emits and data_ram are what Execute does + Escaped, ///< A tagged value reached something other than a move, a store or an emit + Failed, ///< The call fails (as Execute fails it): run it the old way + }; + Outcome status{}; + std::string_view why; + std::vector emits; + std::vector data_ram; + size_t consumed{}; +}; + +/// A dry run of a macro call, as TuringMacroInterpreter::Execute runs it, that follows the list +/// words through it: `params` (the load FIFO) with `params_tagged`, the data RAM words as +/// `read_data_ram(word)` returns them (value, tagged; word < 0xc00), the registers as +/// `read_register(address)` (value, tagged; as the engine's read callback). The call's own +/// stores and emits are kept in `run` and read back from there, as Execute's go to the data RAM +/// and, through the engine, to its registers. Nothing outside `run` changes. +/// +/// A tagged value stays tagged through a move: an ALU operation whose other operand is the +/// identity of the operation and not tagged itself (x + 0, x - 0, x | 0, x ^ 0, x & ~0, +/// x << 0 and the shifts by 0, a merge of no bits), a register, the data RAM, an emit. Any other +/// use (a branch, a predicate, a loop count, an address, a method, any other computation) ends +/// the run as Escaped. Every tagged value is then a list word unchanged: the call takes the same +/// path whatever the words are, and writes them, and only them, where it wrote the tagged values. +template +void RunTagged(std::span code, u32 start, std::span params, + std::span params_tagged, ReadDataRam&& read_data_ram, + ReadRegister&& read_register, std::vector& decoded, + TaggedRun& run) { + using Word = u32; + using Decoded = TuringMacroInterpreter::Decoded; + run = {}; + std::array regs{}; + std::array regs_tagged{}; + Word initialized{}, carry{}, method{}, increment{}; + Word pc = start, loop_count{}, loop_start{}, loop_end{}; + size_t consumed{}; + bool end_next{}, stopped{}; + std::string_view error; + const auto fail = [&](std::string_view message) { + if (error.empty()) + error = message; + }; + bool escaped{}; + const auto escape = [&](std::string_view why) { + if (!escaped) { + escaped = true; + run.why = why; + } + }; + const auto data_ram = [&](Word address) -> std::pair { + for (auto it = run.data_ram.rbegin(); it != run.data_ram.rend(); ++it) { + if (it->address == address) { + return {it->value, it->tagged}; + } + } + return read_data_ram(address); + }; + const auto register_value = [&](Word address) -> std::pair { + // The engine's registers hold what the call emitted to them (FrontApply). + for (auto it = run.emits.rbegin(); it != run.emits.rend(); ++it) { + if (it->address == address) { + return {it->value, it->tagged}; + } + } + return read_register(address); + }; + if (decoded.size() < code.size() / 3) + decoded.resize(code.size() / 3); + constexpr size_t data_ram_size = std::tuple_size_v; + for (Word steps = 0; steps < 1000000 && !escaped; ++steps) { + if (static_cast(pc) * 3 + 2 >= code.size()) { + fail("instruction outside uploaded RAM"); + break; + } + const std::array bits{code[pc * 3 + 2], code[pc * 3 + 1], code[pc * 3]}; + Decoded& d = decoded[pc]; + if (d.raw != bits) + d = TuringMacroInterpreter::Decode(bits); + std::array load{}, alu{}; + std::array load_tagged{}, alu_tagged{}; + for (Word i = 0; i < 2; ++i) { + if (d.load[i]) { + if (consumed >= params.size()) { + fail("parameter FIFO underflow"); + break; + } + load_tagged[i] = consumed < params_tagged.size() && params_tagged[consumed] != 0; + load[i] = params[consumed++]; + } + } + bool operand_tagged{}; + const auto get = [&](Word reg, Word lane) -> Word { + operand_tagged = false; + if (reg < 24) { + if (!(initialized & (1U << reg))) + fail("uninitialized register read"); + operand_tagged = regs_tagged[reg]; + return regs[reg]; + } + switch (reg) { + case 24: + return 0; + case 25: + return Word(std::int32_t(std::int16_t(d.imm[lane]))); + case 26: + return Word(std::int32_t(std::int16_t(d.imm[1 - lane]))); + case 27: + return (d.imm[0] << 16) | d.imm[1]; + case 28: + operand_tagged = load_tagged[0]; + return load[0]; + case 29: + operand_tagged = load_tagged[1]; + return load[1]; + default: + fail("invalid source register"); + return 0; + } + }; + Word mask = 15; + if (d.pred_mode) { + const Word predicate = get(d.pred_reg, 0); + if (operand_tagged) { + escape("a predicate"); + break; + } + mask = predicate ? d.mask_true : d.mask_false; + } + std::array xs{}, ys{}; + std::array xs_tagged{}, ys_tagged{}; + for (Word lane = 0; lane < 2; ++lane) { + if (mask & (1U << lane)) { + xs[lane] = get(d.src_a[lane], lane); + xs_tagged[lane] = operand_tagged; + ys[lane] = get(d.src_b[lane], lane); + ys_tagged[lane] = operand_tagged; + } + } + Word next_pc = pc + 1; + bool branch_cancels_end{}; + for (Word lane = 0; lane < 2 && error.empty() && !escaped; ++lane) { + if (!(mask & (1U << lane))) + continue; + const Word x = xs[lane], y = ys[lane]; + const bool tx = xs_tagged[lane], ty = ys_tagged[lane]; + const Word imm = d.imm[lane]; + Word value{}; + bool tagged{}; + // A tagged operand moved unchanged through the operation: x with y its identity, or + // y with x its identity (`commutes`). + const auto move = [&](Word identity, bool commutes, std::string_view what) { + if (tx && !ty && y == identity) { + tagged = true; + return x; + } + if (commutes && ty && !tx && x == identity) { + tagged = true; + return y; + } + escape(what); + return Word{}; + }; + const auto condition = [&](Word comparison) { + switch (comparison) { + case 17: + case 25: + return std::int32_t(x) < std::int32_t(y); + case 18: + case 26: + return x < y; + case 19: + case 27: + return std::int32_t(x) <= std::int32_t(y); + case 20: + case 28: + return x <= y; + default: + return x == y; + } + }; + const bool any_tagged = tx || ty; + switch (d.op[lane]) { + case 0: + if (any_tagged) { + value = move(0, true, "an addition"); + carry = 0; // x + 0 never carries + } else { + value = x + y; + carry = value < x; + } + break; + case 1: + if (!any_tagged) { + value = x + y + carry; + } else if (carry == 0) { + value = move(0, true, "an addition"); + } else { + escape("an addition"); + } + break; + case 2: + if (any_tagged) { + value = move(0, false, "a subtraction"); + carry = 0; // x - 0 never borrows + } else { + value = x - y; + carry = value > x; + } + break; + case 3: + if (!any_tagged) { + value = x - y - carry; + } else if (carry == 0) { + value = move(0, false, "a subtraction"); + } else { + escape("a subtraction"); + } + break; + case 4: + case 6: { + if (any_tagged) { + escape("a multiplication"); + break; + } + const std::uint64_t a = + d.op[lane] == 4 ? std::uint64_t(std::int64_t(std::int32_t(x))) : x; + const std::uint64_t b = + d.op[lane] == 4 ? std::uint64_t(std::int64_t(std::int32_t(y))) : y; + const auto product = a * b; + value = Word(product); + carry = Word(product >> 32); + break; + } + case 5: + value = carry; + break; + case 7: + if (tx) { + escape("an extended operation"); + break; + } + if (x != 0x1000) + fail("unsupported extended operation"); + break; + case 8: + if (tx) { + escape("a leading zero count"); + break; + } + value = Word(std::countl_zero(x)); + break; + case 9: + case 10: + case 11: + if (any_tagged) { + // Shifts take y's low 5 bits: a tagged x moves by a shift of 0. + if (!ty && (y & 31) == 0) { + value = x; + tagged = true; + } else { + escape("a shift"); + } + } else { + value = d.op[lane] == 9 ? x << (y & 31) + : d.op[lane] == 10 ? x >> (y & 31) + : Word(std::int32_t(x) >> (y & 31)); + } + break; + case 12: + value = any_tagged ? move(~0U, true, "an and") : x & y; + break; + case 13: + if (any_tagged) { + escape("a nand"); + break; + } + value = ~(x & y); + break; + case 14: + value = any_tagged ? move(0, true, "an or") : x | y; + break; + case 15: + value = any_tagged ? move(0, true, "an exclusive or") : x ^ y; + break; + case 16: { + const Word dest = (imm >> 10) & 63, count = (imm >> 5) & 31; + const Word source = imm & 31; + if (dest >= 32) { + fail("invalid merge destination"); + break; + } + const Word bits_mask = (1U << count) - 1; + if (any_tagged) { + // Merging no bits leaves x unchanged; a tagged y never moves whole. + if (tx && !ty && count == 0) { + value = x; + tagged = true; + } else { + escape("a merge"); + } + break; + } + value = (x & ~(bits_mask << dest)) | (((y >> source) & bits_mask) << dest); + break; + } + case 17: + case 18: + case 19: + case 20: + case 21: + if (any_tagged) { + escape("a comparison"); + break; + } + value = condition(d.op[lane]) ? ~0U : 0; + break; + case 22: { + if (any_tagged) { + escape("a register address"); + break; + } + const auto [read, read_tagged] = register_value((x + y) & 0xffff); + value = read; + tagged = read_tagged; + break; + } + case 23: + if (tx) { + escape("a loop count"); + break; + } + if (loop_count || imm < 2) { + fail("invalid loop"); + break; + } + if (x == 0) { + next_pc = pc + imm; + break; + } + loop_count = x - 1; + loop_start = pc; + loop_end = pc + imm - 1; + break; + case 24: + next_pc = pc + (imm & 0x7fff); + break; + case 25: + case 26: + case 27: + case 28: + case 29: + if (any_tagged) { + escape("a branch"); + break; + } + if (condition(d.op[lane]) == bool(imm & 0x8000)) { + const auto offset = std::int32_t(imm << 19) >> 19; + if (offset == -4096) + stopped = true; + else { + next_pc = Word(std::int32_t(pc) + offset); + branch_cancels_end = (imm & 0x4000) != 0; + } + } + break; + case 30: { + if (tx) { + escape("a data RAM address"); + break; + } + if (x >= data_ram_size) { + fail("data RAM read out of bounds"); + break; + } + const auto [read, read_tagged] = data_ram(x); + value = read; + tagged = read_tagged; + break; + } + case 31: + if (tx) { + escape("a data RAM address"); + break; + } + if (x >= data_ram_size) { + fail("data RAM write out of bounds"); + break; + } + run.data_ram.push_back({x, y, ty}); + break; + } + if (escaped) + break; + alu[lane] = value; + alu_tagged[lane] = tagged; + if (d.dst[lane] < 24) { + regs[d.dst[lane]] = value; + regs_tagged[d.dst[lane]] = tagged; + initialized |= 1U << d.dst[lane]; + } else if (d.dst[lane] != 24) + fail("invalid destination register"); + } + if (escaped) + break; + bool output_tagged{}; + const auto output = [&](Word selector) -> Word { + output_tagged = false; + switch (selector) { + case 1: + output_tagged = alu_tagged[0]; + return alu[0]; + case 2: + output_tagged = alu_tagged[1]; + return alu[1]; + case 3: + output_tagged = load_tagged[0]; + return load[0]; + case 4: + output_tagged = load_tagged[1]; + return load[1]; + case 5: + return d.imm[0]; + case 6: + return d.imm[1]; + case 8: + return d.imm[0] >> 12; + case 9: + return d.imm[1] >> 12; + case 10: + return (d.imm[0] << 16) | d.imm[1]; + default: + fail("invalid output selector"); + return 0; + } + }; + for (Word lane = 0; lane < 2 && error.empty() && !escaped; ++lane) { + if (!(mask & (4U << lane))) + continue; + if (d.out_method[lane]) { + const Word address = output(d.out_method[lane]); + if (output_tagged) { + escape("a method"); + break; + } + method = address & 0xfff; + increment = (address >> 12) & 15; + } + if (d.out_emit[lane]) { + const Word value = output(d.out_emit[lane]); + if (error.empty()) { + // The engines' emit refuses macro methods. + if (method >= 0xe00) + fail("unsupported emitted method"); + else + run.emits.push_back({method, value, output_tagged}); + } + method += increment; + } + } + if (escaped || !error.empty()) + break; + if (end_next || stopped) { + run.status = TaggedRun::Outcome::Done; + run.consumed = consumed; + return; + } + end_next = d.end && !branch_cancels_end; + if (loop_count && pc == loop_end) { + --loop_count; + next_pc = loop_start + 1; + } + pc = next_pc; + } + run.consumed = consumed; + if (escaped) { + run.status = TaggedRun::Outcome::Escaped; + return; + } + run.status = TaggedRun::Outcome::Failed; + run.why = error.empty() ? std::string_view{"instruction limit exceeded"} : error; +} + +/// A data RAM word or a register that holds a list word: as long as it holds `value`, the +/// placeholder of word PlaceholderWord(value) of slot PlaceholderSlot(value). Anything that +/// writes it another value ends that (checked where it is used). +struct Holder { + u32 address; + u32 value; +}; + +/// What the macros of one engine's code can read (TuringMacroInterpreter::AnalyzeEffects), by +/// start, for the code generation it was asked for last. +class ReadsCache { +public: + struct Reads { + bool data_ram_known{}; + bool registers_known{}; + std::vector data_ram; + std::vector registers; + }; + + [[nodiscard]] const Reads& Get(std::span code, u64 generation, u32 start) { + if (generation != code_generation || code.data() != code_data) { + index.clear(); + reads.clear(); + code_generation = generation; + code_data = code.data(); + } + if (start >= index.size()) { + index.resize(static_cast(start) + 1, NoEntry); + } + if (index[start] == NoEntry) { + auto effects = TuringMacroInterpreter::AnalyzeEffects(code, start); + reads.push_back({effects.data_ram_reads_known, effects.register_reads_known, + std::move(effects.data_ram_reads), std::move(effects.register_reads)}); + index[start] = static_cast(reads.size() - 1); + } + return reads[index[start]]; + } + +private: + static constexpr u32 NoEntry = ~0U; + std::vector index; + std::vector reads; + u64 code_generation{~u64{0}}; + const u32* code_data{}; +}; + +/// Whether a macro that reads `reads` (any word unless `known`) may read one of `holders`. +[[nodiscard]] inline bool MayRead(bool known, std::span reads, + std::span holders) { + if (holders.empty()) { + return false; + } + if (!known) { + return true; + } + return std::ranges::any_of(holders, [reads](const Holder& holder) { + return std::ranges::binary_search(reads, holder.address); + }); +} + +/// The back drops a hold of `slot` (the rasterizer's Nx2ReleaseListSlot), after what the front +/// sent before; at once while the split is off. +template +void ReleaseSlot(VideoCommon::GpuSplit& split, Rasterizer* rasterizer, u32 slot) { + if (!split.Active()) { + rasterizer->Nx2ReleaseListSlot(slot); + return; + } + split.Defer([rasterizer, slot] { rasterizer->Nx2ReleaseListSlot(slot); }, + VideoCommon::GpuSplit::Writes::Noted); +} + +/// The words of `slots` (three a slot), once the back ran what the front sent before and the +/// host GPU copied them (the rasterizer's Nx2ReadListSlots): the front waits for both. +template +void ReadSlots(VideoCommon::GpuSplit& split, Rasterizer* rasterizer, std::span slots, + std::span words) { + split.Run( + [rasterizer, slot_data = slots.data(), count = slots.size(), out = words.data()] { + rasterizer->Nx2ReadListSlots(std::span{slot_data, count}, + std::span{out, count * SlotWords}); + }, + VideoCommon::DrainSite::ListCopies); +} + +/// The front's holds of the back's slots: one per holder and per call that takes list words. +/// Once the front holds a slot no more, it drops the hold the back gave it the slot with, after +/// everything it sent with the slot's placeholders (in order on the split's ring). +inline std::array& FrontHolds() { + static std::array holds{}; + return holds; +} + +/// A channel's data RAM words that hold list words, and what its 3D macros can read. Its compute +/// engine's front puts list words there; its 3D engine's front checks its macros against them. +struct DataRamWords { + const void* data_ram{}; + std::vector words; + ReadsCache reads_3d; +}; + +/// Every channel's, by data RAM (front thread only; a channel's goes with its compute engine). +inline std::vector& Channels() { + static std::vector channels; + return channels; +} + +[[nodiscard]] inline DataRamWords* FindChannel(const void* data_ram) { + for (DataRamWords& channel : Channels()) { + if (channel.data_ram == data_ram) { + return &channel; + } + } + return nullptr; +} + +[[nodiscard]] inline DataRamWords& GetChannel(const void* data_ram) { + if (DataRamWords* channel = FindChannel(data_ram)) { + return *channel; + } + return Channels().emplace_back(DataRamWords{data_ram, {}, {}}); +} + +/// The channel's compute engine is destroyed. With no channel left (the session ended, and the +/// back's slots went with its rasterizer) the front holds none either. +inline void ForgetChannel(const void* data_ram) { + std::erase_if(Channels(), + [data_ram](const DataRamWords& channel) { return channel.data_ram == data_ram; }); + if (Channels().empty()) { + FrontHolds().fill(0); + } +} + +/// Drops a front hold of `slot`; `release(slot)` once the front holds it no more. +template +void DropHold(u32 slot, Release&& release) { + if (--FrontHolds()[slot] == 0) { + release(slot); + } +} + +/// Drops the holders that no longer hold their placeholder (`value(address)` differs). +template +void Prune(std::vector& holders, Value&& value, Release&& release) { + std::erase_if(holders, [&](const Holder& holder) { + if (value(holder.address) == holder.value) { + return false; + } + DropHold(PlaceholderSlot(holder.value), release); + return true; + }); +} + +/// From now on `address` holds `value`: a list word if `tagged`, else nothing of the front's. +template +void Hold(std::vector& holders, u32 address, u32 value, bool tagged, Release&& release) { + if (tagged) { + ++FrontHolds()[PlaceholderSlot(value)]; // First: an old hold of the slot stays above 0 + } + const auto it = std::ranges::find(holders, address, &Holder::address); + if (it != holders.end()) { + const u32 slot = PlaceholderSlot(it->value); + holders.erase(it); + DropHold(slot, release); + } + if (tagged) { + holders.push_back({address, value}); + } +} + +/// The words of `slots` as `read_slots(slots, words)` returns them (three a slot), by +/// placeholder. +struct SlotWordsRead { + std::vector slots; + std::vector words; + + void Add(u32 placeholder) { + const u32 slot = PlaceholderSlot(placeholder); + if (std::ranges::find(slots, slot) == slots.end()) { + slots.push_back(slot); + } + } + template + void Read(ReadSlots&& read_slots) { + words.assign(slots.size() * SlotWords, 0); + if (!slots.empty()) { + read_slots(std::span{slots}, std::span{words}); + } + } + [[nodiscard]] u32 Word(u32 placeholder) const { + const size_t index = static_cast( + std::ranges::find(slots, PlaceholderSlot(placeholder)) - slots.begin()); + return words[index * SlotWords + PlaceholderWord(placeholder)]; + } +}; + +/// Replaces the placeholders the data RAM words `holders` still hold by their words, read back +/// from the slots (`read_slots` waits for the back and the host GPU), and drops the holders. +/// Returns how many slots it read. +template +u32 ConcretizeDataRam(std::vector& holders, std::span data_ram, ReadSlots&& read_slots, + Release&& release) { + SlotWordsRead read; + for (const Holder& holder : holders) { + read.Add(holder.value); + } + read.Read(read_slots); + for (const Holder& holder : holders) { + if (data_ram[holder.address] == holder.value) { + data_ram[holder.address] = read.Word(holder.value); + } + DropHold(PlaceholderSlot(holder.value), release); + } + holders.clear(); + return static_cast(read.slots.size()); +} + +/// What the front knows of a compute engine's inline upload as the back runs it +/// (Upload::State): the words the data methods sent since the last exec, and the size and layout +/// the exec started it with. +struct UploadState { + u32 words{}; + u32 size{}; + bool linear{}; +}; + +/// The compute class's inline upload methods (KeplerCompute::Regs::upload and the two after). +inline constexpr u32 UploadLineLength = 0x60; +inline constexpr u32 UploadLineCount = 0x61; +inline constexpr u32 UploadAddressHigh = 0x62; +inline constexpr u32 UploadAddressLow = 0x63; +inline constexpr u32 UploadExec = 0x6c; +inline constexpr u32 UploadData = 0x6d; +/// The inline QMD's grid words (LOAD_INLINE_QMD_DATA, QMD words 12-14). +inline constexpr u32 InlineQmdGridFirst = 0xc8 + 12; +inline constexpr u32 InlineQmdGridLast = 0xc8 + 14; + +/// As Upload::State::ProcessExec and ProcessData count an inline upload: `method` with +/// `argument` sent to the engine, `reg(method)` the registers as they were before it. +template +void NoteUploadMethod(UploadState& upload, u32 method, u32 argument, Register&& reg) { + if (method == UploadExec) { + upload.words = 0; + upload.size = reg(UploadLineLength) * reg(UploadLineCount); + upload.linear = (argument & 1) != 0; + } else if (method == UploadData) { + ++upload.words; + } +} + +/// Why the front read list words after all (the counters). +enum class Reason : u32 { + Escaped, ///< A macro used a tagged value otherwise than moving it (RunTagged) + Failed, ///< The dry run failed: the call runs the old way, as it fails there + Sink, ///< A tagged value reached another register, or an upload the back can't place + Upload, ///< A macro that takes list words loads macro code or binds macros + Macro3d, ///< A 3D macro may read a data RAM word that holds one + Stop, ///< The split stopped + Count, +}; + +/// One compute engine's front while list words are about (KeplerCompute::FrontRunMacro): the +/// registers that hold list words, the call whose parameters are list words, and the caches of +/// the dry runs. `Context` is the engine's, for one call: +/// - std::span code; u64 generation; u32 start: the call's macro; +/// - std::vector& params; std::span data_ram; std::span registers; +/// - UploadState upload: before the call; +/// - std::optional DeviceWord(u64 gpu_address): the device address of the 4 bytes there, +/// if they are mapped, on one page, as the back will find them; +/// - void Emit(u32 method, u32 value): the engine's emit (FrontApply and the split); +/// - void SendOwed(u64 device, u32 value): the back patches the slot word that `value` is the +/// placeholder of into the constant buffers compute dispatches bind there; +/// - void Release(u32 slot): the back drops the front's hold of the slot; +/// - void ReadSlots(std::span slots, std::span words): the words (waits); +/// - void SetRegister(u32 method, u32 value): a concretized register, on the front and the back. +class ComputeFront { +public: + /// DmaPusher, at a list of `count` words on GPU-written pages that are all the parameters of + /// the call about to run: whether a dry run of the call with them as tagged parameters leaves + /// them only where the back takes them over. If so the call takes `slot`'s placeholders as + /// its parameters (the front holds the slot for it) and runs tagged (Run). + template + [[nodiscard]] bool PrepareCall(Context& ctx, DataRamWords& channel, u32 slot, u32 count) { + // Holders the macros the front ran untagged wrote over give their slots back. + const auto release = [&ctx](u32 held) { ctx.Release(held); }; + Prune(channel.words, [&ctx](u32 address) { return ctx.data_ram[address]; }, release); + Prune(registers, [&ctx](u32 address) { return ctx.registers[address]; }, release); + std::vector params(count); + for (u32 k = 0; k < count; ++k) { + params[k] = Placeholder(slot, k); + } + TaggedRun run; + DryRun(ctx, channel, params, true, run); + if (run.status != TaggedRun::Outcome::Done || !Places(ctx, run)) { + return false; + } + if (pending.active) [[unlikely]] { + DropPending(ctx); + } + pending = Pending{slot, count, true}; + ++FrontHolds()[slot]; + return true; + } + + /// The call `ctx` describes, at its last parameter. If it takes list words, or may read a + /// data RAM word or register that holds one, runs it tagged and returns true. Otherwise, and + /// where its list words go elsewhere than the back follows them (it concretizes every list + /// word about first: `concretized`, `reason`), the caller runs it as before (false). + template + [[nodiscard]] bool Run(Context& ctx, DataRamWords& channel, bool& concretized, Reason& reason) { + concretized = false; + const bool takes = + pending.active && pending.count == ctx.params.size() && TakesPlaceholders(ctx.params); + if (pending.active && !takes) [[unlikely]] { + DropPending(ctx); // Not the call it was prepared for + } + if (!takes) { + if (channel.words.empty() && registers.empty()) { + return false; + } + if (!MayReadHeld(ctx, channel)) { + return false; + } + const auto release = [&ctx](u32 slot) { ctx.Release(slot); }; + Prune(channel.words, [&ctx](u32 address) { return ctx.data_ram[address]; }, release); + Prune(registers, [&ctx](u32 address) { return ctx.registers[address]; }, release); + if (!MayReadHeld(ctx, channel)) { + return false; + } + } + TaggedRun run; + DryRun(ctx, channel, ctx.params, takes, run); + if (run.status == TaggedRun::Outcome::Done && Places(ctx, run)) { + Commit(ctx, channel, run); + if (takes) { + DropPending(ctx); + } + return true; + } + reason = run.status == TaggedRun::Outcome::Escaped ? Reason::Escaped + : run.status == TaggedRun::Outcome::Failed ? Reason::Failed + : place_reason; + ConcretizeAll(ctx, channel); + concretized = true; + return false; + } + + /// Replaces every list word the channel's data RAM, this engine's registers and the pending + /// call's parameters hold by the word itself, read back from its slot. + template + void ConcretizeAll(Context& ctx, DataRamWords& channel) { + const auto release = [&ctx](u32 slot) { ctx.Release(slot); }; + const bool params = + pending.active && pending.count == ctx.params.size() && TakesPlaceholders(ctx.params); + SlotWordsRead read; + for (const Holder& holder : channel.words) { + read.Add(holder.value); + } + for (const Holder& holder : registers) { + read.Add(holder.value); + } + if (params) { + read.Add(Placeholder(pending.slot, 0)); + } + read.Read([&ctx](std::span slots, std::span words) { + ctx.ReadSlots(slots, words); + }); + for (const Holder& holder : channel.words) { + if (ctx.data_ram[holder.address] == holder.value) { + ctx.data_ram[holder.address] = read.Word(holder.value); + } + DropHold(PlaceholderSlot(holder.value), release); + } + channel.words.clear(); + for (const Holder& holder : registers) { + if (holder.address < ctx.registers.size() && + ctx.registers[holder.address] == holder.value) { + ctx.SetRegister(holder.address, read.Word(holder.value)); + } + DropHold(PlaceholderSlot(holder.value), release); + } + registers.clear(); + if (params) { + for (u32 k = 0; k < pending.count; ++k) { + ctx.params[k] = read.Word(Placeholder(pending.slot, k)); + } + } + DropPending(ctx); + } + + /// The registers that hold list words. + [[nodiscard]] std::span Registers() const { + return registers; + } + +private: + struct Pending { + u32 slot{}; + u32 count{}; + bool active{}; + }; + + [[nodiscard]] bool TakesPlaceholders(std::span params) const { + for (u32 k = 0; k < pending.count; ++k) { + if (params[k] != Placeholder(pending.slot, k)) { + return false; + } + } + return true; + } + + template + void DropPending(Context& ctx) { + if (!pending.active) { + return; + } + pending.active = false; + DropHold(pending.slot, [&ctx](u32 slot) { ctx.Release(slot); }); + } + + /// Whether the call's macro may read a data RAM word or register that holds a list word. + template + [[nodiscard]] bool MayReadHeld(Context& ctx, const DataRamWords& channel) { + const auto& reads = cache.Get(ctx.code, ctx.generation, ctx.start); + return Nx2ListIndirect::MayRead(reads.data_ram_known, reads.data_ram, channel.words) || + Nx2ListIndirect::MayRead(reads.registers_known, reads.registers, registers); + } + + /// The dry run of the call with `params` (tagged if `tagged`), the data RAM and the + /// registers with their holders. + template + void DryRun(Context& ctx, const DataRamWords& channel, std::span params, bool tagged, + TaggedRun& run) { + params_tagged.assign(params.size(), static_cast(tagged ? 1 : 0)); + RunTagged( + ctx.code, ctx.start, params, std::span{params_tagged}, + [&](u32 word) -> std::pair { + const u32 value = ctx.data_ram[word]; + const auto it = std::ranges::find(channel.words, word, &Holder::address); + return {value, it != channel.words.end() && it->value == value}; + }, + [&](u32 address) -> std::pair { + if (address >= ctx.registers.size()) { + return {0U, false}; + } + const u32 value = ctx.registers[address]; + const auto it = std::ranges::find(registers, address, &Holder::address); + return {value, it != registers.end() && it->value == value}; + }, + decoded, run); + } + + /// Whether every tagged value the run emits lands where the back takes it over: an inline + /// QMD grid word (the back dispatches from the slot) or a word of a linear one-line inline + /// upload the front can place (the back patches it into constant buffers). The data RAM + /// keeps the others. Notes where each such upload word lands in `owed`. + template + [[nodiscard]] bool Places(Context& ctx, const TaggedRun& run) { + owed.assign(run.emits.size(), 0); + UploadState upload = ctx.upload; + const auto reg = [&](size_t before, u32 method) { + for (size_t i = before; i-- > 0;) { + if (run.emits[i].address == method) { + return run.emits[i].value; + } + } + return method < ctx.registers.size() ? ctx.registers[method] : 0U; + }; + for (size_t i = 0; i < run.emits.size(); ++i) { + const TaggedWrite& emit = run.emits[i]; + if (emit.address >= 0x45 && emit.address <= 0x48) { + place_reason = Reason::Upload; // FrontApply loads code and bindings + return false; + } + if (emit.address == UploadData && emit.tagged) { + // As Upload::State::ProcessData writes a word of a linear upload of one line. + if (!upload.linear || reg(i, UploadLineCount) != 1 || + u64{upload.words} * 4 + 4 > upload.size) { + place_reason = Reason::Sink; + return false; + } + const u64 destination = + ((u64{reg(i, UploadAddressHigh)} << 32) | reg(i, UploadAddressLow)) + + u64{upload.words} * 4; + const std::optional device = ctx.DeviceWord(destination); + if (!device) { + place_reason = Reason::Sink; + return false; + } + owed[i] = *device; + } else if (emit.tagged && + (emit.address < InlineQmdGridFirst || emit.address > InlineQmdGridLast)) { + place_reason = Reason::Sink; + return false; + } + NoteUploadMethod(upload, emit.address, emit.value, + [&](u32 method) { return reg(i, method); }); + } + return true; + } + + /// Applies the run: its data RAM stores, its emits through the engine, and where list words + /// are now. + template + void Commit(Context& ctx, DataRamWords& channel, const TaggedRun& run) { + const auto release = [&ctx](u32 slot) { ctx.Release(slot); }; + for (const TaggedWrite& store : run.data_ram) { + ctx.data_ram[store.address] = store.value; + Hold(channel.words, store.address, store.value, store.tagged, release); + } + for (size_t i = 0; i < run.emits.size(); ++i) { + const TaggedWrite& emit = run.emits[i]; + ctx.Emit(emit.address, emit.value); + if (emit.address == UploadData && emit.tagged) { + ctx.SendOwed(owed[i], emit.value); + } + if (emit.tagged || !registers.empty()) { + Hold(registers, emit.address, emit.value, emit.tagged, release); + } + } + } + + std::vector registers; + Pending pending; + ReadsCache cache; + std::vector decoded; + std::vector params_tagged; + std::vector owed; + Reason place_reason{}; +}; + +} // namespace Tegra::Nx2ListIndirect diff --git a/src/video_core/nx2_list_indirect_stats.h b/src/video_core/nx2_list_indirect_stats.h new file mode 100644 index 0000000000..7940b02151 --- /dev/null +++ b/src/video_core/nx2_list_indirect_stats.h @@ -0,0 +1,237 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "common/common_types.h" +#include "common/logging.h" +#include "video_core/nx2_list_indirect.h" +#include "video_core/nx2_profile.h" + +// TEST: what Nx2ListIndirect did, logged every 10 s by the front with EDEN_NX2_GPU_PROFILE or +// EDEN_NX2_LIST_INDIRECT_CHECK: +// - the GPU-written lists the front took without waiting for their writers, by what takes them +// (the compute macro method), and the lists it read as before, by why; +// - the times it read list words back after all (waits the profile counts as "list copy +// writers"), how long it waited, and why; +// - on the back: snapshots, indirect dispatches, grids read back, owed words and the dispatches +// whose constant buffers got them. +// EDEN_NX2_LIST_INDIRECT_CHECK also has the front read every list it takes as before (it waits for +// the list's writers, as the old path does), and the back compare what the host GPU used, read +// back, with what the old path would have used: the slot's words with the list's (snapshot), the +// group counts the dispatch took with the grid the old path dispatches (dispatch), the words +// copied over a constant buffer's placeholders with the list's (patch), the words read back with +// the list's (read), and guest memory with the placeholder an inline upload just wrote there +// (owed: the patch takes over only words guest memory holds). Differences must stay 0; the first +// ones are logged. It counts the dispatches the old path would have skipped (an empty grid: the +// indirect one dispatches no groups) and widths over 31 bits (the old path drops the top bit), +// which differ only on the CPU. +namespace Tegra::Nx2ListIndirect::Stats { + +// A plain flag: checked at every list the host GPU takes over. +inline bool g_check = std::getenv("EDEN_NX2_LIST_INDIRECT_CHECK") != nullptr; + +/// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads +/// nx2_switches.txt, so a switch set only there takes effect from that point on. +inline void ReadSwitch() { + g_check = std::getenv("EDEN_NX2_LIST_INDIRECT_CHECK") != nullptr; +} + +[[nodiscard]] inline bool Enabled() { + return g_check || VideoCommon::Nx2Profile::Enabled(); +} + +/// Why a GPU-written list took the old path (the front read its words). +enum class Kept : u32 { + Off, ///< EDEN_NX2_LIST_INDIRECT=0, the split without macros on its front, no slots + NotCall, ///< Not all the parameters of a compute macro call (3D lists, commands, ...) + Placement, ///< Not on one mapped page, or map changes the back has yet to run + NoSlot, ///< Every slot held + Refused, ///< The engine's front or the call's dry run refused it + Count, +}; +inline constexpr std::array(Kept::Count)> kept_names{ + "off", + "not all of a compute macro call's parameters", + "not on one mapped page or map changes queued", + "no free slot", + "the call's dry run refused", +}; + +inline constexpr std::array(Reason::Count)> reason_names{ + "a macro used one", "a dry run failed", "one went elsewhere", + "macro code or binding", "a 3D macro may read one", "the split stopped", +}; + +/// The back's comparisons (EDEN_NX2_LIST_INDIRECT_CHECK). +enum class Check : u32 { Snapshot, Dispatch, Patch, Read, Owed, Count }; +inline constexpr std::array(Check::Count)> check_names{ + "snapshot", "dispatch", "patch", "read", "owed"}; + +struct Counters { + // The front's. + std::map> taken; ///< By macro method: lists, words + std::array(Kept::Count)> kept{}; + std::array(Reason::Count)> concretized{}; + u64 reads{}; ///< Waits for list words read back + u64 slots_read{}; + u64 read_ns{}; + std::chrono::steady_clock::time_point last_log{std::chrono::steady_clock::now()}; + // The back's. + std::atomic snapshots{}; + std::atomic slot_dispatches{}; + std::atomic grids_read{}; ///< Grids of several slots' words, read back to dispatch + std::atomic owed{}; + std::atomic owed_dropped{}; ///< Owed words guest memory no longer held + std::atomic patched_binds{}; + std::array, static_cast(Check::Count)> compared{}; + std::array, static_cast(Check::Count)> differ{}; + std::atomic unchecked{}; ///< Comparisons dropped: the check buffer was full + std::atomic empty_grids{}; + std::atomic wide_grids{}; + std::atomic differences_logged{}; +}; + +[[nodiscard]] inline Counters& Get() { + static Counters counters; + return counters; +} + +/// The front: a GPU-written list took the old path. +inline void NoteKept(Kept why) { + if (Enabled()) [[unlikely]] { + ++Get().kept[static_cast(why)]; + } +} + +/// The front: a list of `words` words the call of compute macro method `method` takes. +inline void NoteTaken(u32 method, u32 words) { + if (Enabled()) [[unlikely]] { + auto& entry = Get().taken[method]; + ++entry[0]; + entry[1] += words; + } +} + +/// The front read every list word about back after all. +inline void NoteConcretized(Reason reason) { + if (Enabled()) [[unlikely]] { + ++Get().concretized[static_cast(reason)]; + } +} + +/// The front waited `ns` for the words of `slots` slots. +inline void NoteRead(size_t slots, u64 ns) { + if (Enabled()) [[unlikely]] { + ++Get().reads; + Get().slots_read += slots; + Get().read_ns += ns; + } +} + +[[nodiscard]] inline u64 NowNs() { + return static_cast(std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()) + .count()); +} + +/// Any thread: `counter` of the back's. +inline void Count(std::atomic& counter, u64 amount = 1) { + if (Enabled()) [[unlikely]] { + counter.fetch_add(amount, std::memory_order_relaxed); + } +} + +/// The back: a comparison of `check` for `slot`, the bytes the host GPU used (`got`) against the +/// ones the old path would have (`expected`). +inline void NoteCompared(Check check, u32 slot, std::span got, + std::span expected) { + auto& counters = Get(); + counters.compared[static_cast(check)].fetch_add(1, std::memory_order_relaxed); + if (std::ranges::equal(got, expected)) { + return; + } + counters.differ[static_cast(check)].fetch_add(1, std::memory_order_relaxed); + if (counters.differences_logged.fetch_add(1, std::memory_order_relaxed) < 32) { + LOG_ERROR(HW_GPU, "Nx2 list indirect check: {} of slot {} differs: {:02x}, expected {:02x}", + check_names[static_cast(check)], slot, fmt::join(got, " "), + fmt::join(expected, " ")); + } +} + +/// The front, every 10 s. +inline void MaybeLog() { + auto& counters = Get(); + const auto now = std::chrono::steady_clock::now(); + const double seconds = std::chrono::duration(now - counters.last_log).count(); + if (seconds < 10.0) { + return; + } + counters.last_log = now; + u64 taken = 0; + u64 taken_words = 0; + std::string by_method; + for (const auto& [method, entry] : counters.taken) { + taken += entry[0]; + taken_words += entry[1]; + by_method += fmt::format("{}compute {:#x} {} ({} words)", by_method.empty() ? "" : ", ", + method, entry[0], entry[1]); + } + u64 kept = 0; + std::string why_kept; + for (size_t i = 0; i < counters.kept.size(); ++i) { + kept += counters.kept[i]; + why_kept += + fmt::format("{}{} {}", why_kept.empty() ? "" : ", ", kept_names[i], counters.kept[i]); + } + std::string why_read; + for (size_t i = 0; i < counters.concretized.size(); ++i) { + why_read += fmt::format("{}{} {}", why_read.empty() ? "" : ", ", reason_names[i], + counters.concretized[i]); + } + const auto take = [](std::atomic& counter) { + return counter.exchange(0, std::memory_order_relaxed); + }; + LOG_INFO(HW_GPU, + "list indirect {:.1f}s: GPU-written lists taken without waiting {} ({} words): {} | " + "read as before {}: {} | list words read back {} times ({} slots, {:.1f} ms), all " + "of them after: {} | back: snapshots {}, indirect dispatches {}, grids read back {}, " + "owed words {} ({} overwritten), dispatches given owed words {}", + seconds, taken, taken_words, by_method.empty() ? "-" : by_method, kept, why_kept, + counters.reads, counters.slots_read, static_cast(counters.read_ns) / 1e6, + why_read, take(counters.snapshots), take(counters.slot_dispatches), + take(counters.grids_read), take(counters.owed), take(counters.owed_dropped), + take(counters.patched_binds)); + if (g_check) { + std::string checks; + for (size_t i = 0; i < counters.compared.size(); ++i) { + checks += fmt::format("{}{} {} ({} differ)", checks.empty() ? "" : ", ", check_names[i], + take(counters.compared[i]), take(counters.differ[i])); + } + LOG_INFO(HW_GPU, + "list indirect check {:.1f}s: compared {}; unchecked {} | dispatches the old " + "path skips (empty grid) {}, widths over 31 bits {}", + seconds, checks, take(counters.unchecked), take(counters.empty_grids), + take(counters.wide_grids)); + } + counters.taken.clear(); + counters.kept = {}; + counters.concretized = {}; + counters.reads = 0; + counters.slots_read = 0; + counters.read_ns = 0; +} + +} // namespace Tegra::Nx2ListIndirect::Stats diff --git a/src/video_core/nx2_list_stats.h b/src/video_core/nx2_list_stats.h new file mode 100644 index 0000000000..47532469de --- /dev/null +++ b/src/video_core/nx2_list_stats.h @@ -0,0 +1,580 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "common/common_types.h" +#include "common/logging.h" +#include "video_core/nx2.h" + +// EDEN_NX2_LIST_STATS: what the front's reads of command lists on GPU-written pages +// (DmaPusher::ReadGpuWrittenCommands) wait for. For each list its size; for each of its pages how +// BufferCache::Nx2ReadListCopy served it, the GPU work that last wrote the page (a dispatch's +// shader, a copy engine copy's source, or draws) and whether the copy held other bytes than guest +// memory; for each copy the front waited for: how long, how long before the back had recorded it, +// how much the back still had to run (and of that, not yet published) and whether other channels +// had lists queued. Every 10 s the front logs the counts. +// +// EDEN_NX2_LIST_VERIFY: each list read so is read again as the front read lists before it tracked +// the words GPU work wrote (after every dispatch and copy sent so far, from the pages' whole +// copies, or drained and flushed) and, when the two differ, drained and flushed as well, which +// tells the stale read. Differences are counted and the first ones logged; the front parses the +// words read last. +namespace Tegra::Nx2ListStats { + +// Plain flags: checked at every GPU-written list and buffer write mark. +inline bool g_enabled = std::getenv("EDEN_NX2_LIST_STATS") != nullptr; +inline bool g_verify = std::getenv("EDEN_NX2_LIST_VERIFY") != nullptr; + +/// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads +/// nx2_switches.txt, so a switch set only there takes effect from that point on. +inline void ReadSwitches() { + g_enabled = std::getenv("EDEN_NX2_LIST_STATS") != nullptr; + g_verify = std::getenv("EDEN_NX2_LIST_VERIFY") != nullptr; +} + +[[nodiscard]] inline bool Any() { + return g_enabled || g_verify; +} + +[[nodiscard]] inline u64 NowNs() { + return static_cast(std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()) + .count()); +} + +inline constexpr u64 PageBytes = u64{1} << Nx2WrittenPageBits; + +/// How BufferCache::Nx2ReadListCopy served a list's bytes on one page. +enum class Served : u32 { + PageSynced, ///< From guest memory: the front flushed the page, no GPU work wrote it since + WordsClean, ///< From guest memory: no GPU work wrote the list's words since the flush + CopyReady, ///< From the page's copy, which the GPU had finished + CopyWaited, ///< From the page's copy, once the GPU finished it + Unknown, ///< Nowhere: the front drains and flushes + Count, +}; + +inline constexpr std::array(Served::Count)> served_names{ + "page synced", "words clean", "copy ready", "copy waited", "unknown", +}; + +/// The GPU work that last wrote a command list page (BufferCache::Nx2NoteListWrite). +enum class Writer : u32 { + None, ///< No write noted since the stats began + Dispatch, ///< A compute dispatch: the id is its shader's hash + Copy, ///< A copy engine copy: the id is its source GPU address + Draw, ///< Draws and the other GPU writes + Count, +}; + +inline constexpr std::array(Writer::Count)> writer_names{ + "none", + "dispatch", + "copy", + "draw", +}; + +struct WriterTag { + Writer writer{Writer::Draw}; + u64 id{}; +}; + +/// The back: the dispatch or copy the GPU writes noted now come from. +inline thread_local WriterTag t_writer{}; + +/// The back, around a dispatch or a copy engine copy: tags the GPU writes noted meanwhile. +class WriterScope { +public: + explicit WriterScope(Writer writer, u64 id) : previous{t_writer} { + t_writer = WriterTag{writer, id}; + } + ~WriterScope() { + t_writer = previous; + } + WriterScope(const WriterScope&) = delete; + WriterScope& operator=(const WriterScope&) = delete; + +private: + WriterTag previous; +}; + +/// What Nx2ReadListCopy told about its last read where its result does not (NoteServed). +struct ServedDetail { + bool valid{}; + Served served{Served::Unknown}; + u64 wait_start_ns{}; ///< CopyWaited: when the front started to wait + u64 waited_ns{}; +}; +inline thread_local ServedDetail t_served{}; + +/// The front, in Nx2ReadListCopy: how it served the read. +inline void NoteServed(Served served, u64 wait_start_ns = 0, u64 waited_ns = 0) { + if (Any()) { + t_served = ServedDetail{true, served, wait_start_ns, waited_ns}; + } +} + +/// The front, while EDEN_NX2_LIST_VERIFY reads a list as before: Nx2ReadListCopy serves guest +/// memory only for pages flushed and written by no GPU work since, else the page's whole copy. +inline thread_local bool t_reference{}; + +class ReferenceScope { +public: + ReferenceScope() { + t_reference = true; + } + ~ReferenceScope() { + t_reference = false; + } + ReferenceScope(const ReferenceScope&) = delete; + ReferenceScope& operator=(const ReferenceScope&) = delete; +}; + +inline constexpr size_t ChannelSlots = 16; +/// Command lists submitted to the GPU thread (ThreadManager::SubmitList) that it has not taken +/// yet, by channel. +inline std::array, ChannelSlots> g_queued_lists{}; + +[[nodiscard]] inline size_t ChannelSlot(s32 channel) { + return static_cast(static_cast(channel)) % ChannelSlots; +} + +/// Any thread: a list of `channel` was queued for the GPU thread (`delta` 1) or taken (-1). +inline void NoteListQueued(s32 channel, s32 delta) { + if (g_enabled) { + g_queued_lists[ChannelSlot(channel)].fetch_add(delta, std::memory_order_relaxed); + } +} + +/// Whether lists of channels other than `channel` wait for the GPU thread. +[[nodiscard]] inline bool OtherChannelsQueued(s32 channel) { + const size_t own = ChannelSlot(channel); + for (size_t slot = 0; slot < ChannelSlots; ++slot) { + if (slot != own && g_queued_lists[slot].load(std::memory_order_relaxed) > 0) { + return true; + } + } + return false; +} + +struct Waits { + u64 reads{}; + u64 waits{}; + u64 wait_ns{}; + + void Add(bool waited, u64 ns) { + ++reads; + if (waited) { + ++waits; + wait_ns += ns; + } + } +}; + +struct PageInfo { + WriterTag writer{Writer::None, 0}; + u32 write_begin{}; ///< The last GPU write's bytes on the page + u32 write_end{}; + u64 write_size{}; ///< The last GPU write's whole size + u64 copy_ns{}; ///< When the back recorded the page's last copy + Waits waits; ///< This period's reads + u32 list_begin{~u32{0}}; ///< The bytes of the page this period's lists read + u32 list_end{}; +}; + +struct WriterStats { + WriterTag writer; + Waits waits; ///< This period's reads of the pages it wrote last + u64 writes{}; ///< This period's GPU writes to command list pages + u64 bytes{}; ///< Their bytes on those pages +}; + +/// How a list differed from its read as before (EDEN_NX2_LIST_VERIFY). +enum class Difference : u32 { + OldStale, ///< The drained read matches the new read: the read as before was stale + NewStale, ///< The drained read matches the read as before: the new read was stale + AllDiffer, ///< All three differ + Drained, ///< The read as before drained and flushed: the new read was stale + Count, +}; + +inline constexpr std::array(Difference::Count)> difference_names{ + "the read as before was stale", + "the new read was stale", + "all three differ", + "the new read differs from the drained one", +}; + +/// One page of the list the front reads, for EDEN_NX2_LIST_VERIFY's log. +struct ListPart { + u64 page{}; + u32 offset{}; + u32 size{}; + Served served{Served::Unknown}; +}; + +inline constexpr std::array list_size_limits{16, 64, 256, 1024, 4096}; +inline constexpr std::array wait_limits_us{100, 250, 500, 1000}; + +/// The counts of one logging period. +struct Period { + u64 lists{}; + u64 list_words{}; + std::array list_sizes{}; + std::array(Served::Count)> served{}; + std::array(Writer::Count)> by_writer{}; + u64 after_sync{}; ///< Reads that first waited for the work before the last sync + u64 wait_ns{}; + u64 wait_max_ns{}; + std::array wait_sizes{}; + u64 copies_compared{}; + u64 copies_equal{}; + u64 waits_backlog{}; ///< Copy waits the back had work for + u64 backlog_words{}; + u64 waits_unpublished{}; ///< Copy waits with work the back did not see yet + u64 unpublished_words{}; + u64 waits_other_channels{}; + u64 waits_recorded{}; ///< Copy waits whose copy's record time is known + u64 since_recorded_ns{}; + + u64 verified{}; + u64 verified_drained{}; ///< Read as before by draining and flushing + u64 differ{}; + u64 differ_words{}; + std::array(Difference::Count)> differences{}; +}; + +struct State { + std::mutex mutex; + std::unordered_map pages; + std::unordered_map writers; + std::vector parts; ///< The pages of the list the front reads + Period period; + u64 verified_total{}; + u64 differ_total{}; + u32 differ_logged{}; + std::chrono::steady_clock::time_point last_log{std::chrono::steady_clock::now()}; +}; + +inline State& Get() { + static State state; + return state; +} + +[[nodiscard]] inline u64 WriterKey(const WriterTag& tag) { + return (static_cast(tag.writer) << 60) ^ tag.id; +} + +[[nodiscard]] inline std::string WriterText(const WriterTag& tag) { + switch (tag.writer) { + case Writer::Dispatch: + return fmt::format("dispatch {:#018x}", tag.id); + case Writer::Copy: + return fmt::format("copy from {:#x}", tag.id); + default: + return writer_names[static_cast(tag.writer)]; + } +} + +/// The index of the first of `limits` `value` does not exceed, else limits.size(). +[[nodiscard]] inline size_t Bucket(u64 value, std::span limits) { + size_t bucket = 0; + while (bucket < limits.size() && value > limits[bucket]) { + ++bucket; + } + return bucket; +} + +[[nodiscard]] inline double Ms(u64 ns) { + return static_cast(ns) / 1e6; +} + +[[nodiscard]] inline double Average(u64 total, u64 count, double unit) { + return count == 0 ? 0.0 : static_cast(total) / static_cast(count) / unit; +} + +/// The back: GPU work wrote [device_addr, device_addr + size) over the command list page at +/// `page_addr` (BufferCache::Nx2NoteListWrite). +inline void NoteWrite(u64 page_addr, u64 device_addr, u64 size) { + const u64 begin = (std::max)(device_addr, page_addr) - page_addr; + const u64 end = (std::min)(device_addr + size, page_addr + PageBytes) - page_addr; + const WriterTag tag = t_writer; + State& state = Get(); + std::scoped_lock lock{state.mutex}; + PageInfo& page = state.pages[page_addr]; + page.writer = tag; + page.write_begin = static_cast(begin); + page.write_end = static_cast(end); + page.write_size = size; + WriterStats& stats = state.writers[WriterKey(tag)]; + stats.writer = tag; + ++stats.writes; + stats.bytes += end - begin; +} + +/// The back recorded the copy of the command list page at `page_addr` (Nx2CopyListPages). +inline void NoteCopy(u64 page_addr) { + const u64 now = NowNs(); + State& state = Get(); + std::scoped_lock lock{state.mutex}; + state.pages[page_addr].copy_ns = now; +} + +inline void LogStats(const State& state, double seconds) { + const Period& counts = state.period; + const auto writer_reads = [&counts](Writer kind) { + const Waits& entry = counts.by_writer[static_cast(kind)]; + return fmt::format("{} {}/{}/{:.1f}", writer_names[static_cast(kind)], entry.reads, + entry.waits, Ms(entry.wait_ns)); + }; + const u64 copy_waits = counts.served[static_cast(Served::CopyWaited)]; + LOG_INFO( + HW_GPU, + "list stats {:.1f}s: {} lists of {} words (up to 16 words {}, 64 {}, 256 {}, 1024 {}, " + "4096 {}, more {}) | page reads: page synced {}, words clean {}, copy ready {}, copy " + "waited {}, unknown {}; after a sync wait {} | copy waits {:.1f} ms, average {:.0f} " + "us, max {:.0f} us (up to 100 us {}, 250 {}, 500 {}, 1000 {}, more {}) | by the " + "page's last writer (reads/waits/ms): {}, {}, {}, {} | copies equal to guest memory " + "{} of {} | at copy waits: the back had work {} (average {:.0f} words), not yet " + "published {} (average {:.0f} words), other channels had lists queued {}, the copy " + "was recorded {:.0f} us before on average ({} known)", + seconds, counts.lists, counts.list_words, counts.list_sizes[0], counts.list_sizes[1], + counts.list_sizes[2], counts.list_sizes[3], counts.list_sizes[4], counts.list_sizes[5], + counts.served[0], counts.served[1], counts.served[2], counts.served[3], counts.served[4], + counts.after_sync, Ms(counts.wait_ns), Average(counts.wait_ns, copy_waits, 1e3), + static_cast(counts.wait_max_ns) / 1e3, counts.wait_sizes[0], counts.wait_sizes[1], + counts.wait_sizes[2], counts.wait_sizes[3], counts.wait_sizes[4], + writer_reads(Writer::Dispatch), writer_reads(Writer::Copy), writer_reads(Writer::Draw), + writer_reads(Writer::None), counts.copies_equal, counts.copies_compared, + counts.waits_backlog, Average(counts.backlog_words, counts.waits_backlog, 1.0), + counts.waits_unpublished, Average(counts.unpublished_words, counts.waits_unpublished, 1.0), + counts.waits_other_channels, Average(counts.since_recorded_ns, counts.waits_recorded, 1e3), + counts.waits_recorded); + + std::vector writers; + for (const auto& entry : state.writers) { + writers.push_back(&entry.second); + } + std::ranges::sort(writers, [](const WriterStats* lhs, const WriterStats* rhs) { + return std::tie(lhs->waits.wait_ns, lhs->waits.reads, lhs->writes) > + std::tie(rhs->waits.wait_ns, rhs->waits.reads, rhs->writes); + }); + std::string writers_text; + for (size_t i = 0; i < writers.size() && i < 8; ++i) { + const WriterStats& stats = *writers[i]; + writers_text += fmt::format( + "{}{}: {} reads, {} waits in {:.1f} ms, {} writes of {} bytes on list pages", + writers_text.empty() ? "" : "; ", WriterText(stats.writer), stats.waits.reads, + stats.waits.waits, Ms(stats.waits.wait_ns), stats.writes, stats.bytes); + } + if (!writers_text.empty()) { + LOG_INFO(HW_GPU, "list stats writers: {}", writers_text); + } + + std::vector> pages; + for (const auto& entry : state.pages) { + if (entry.second.waits.reads != 0) { + pages.emplace_back(entry.first, &entry.second); + } + } + std::ranges::sort(pages, [](const auto& lhs, const auto& rhs) { + return std::tie(lhs.second->waits.wait_ns, lhs.second->waits.reads) > + std::tie(rhs.second->waits.wait_ns, rhs.second->waits.reads); + }); + std::string pages_text; + for (size_t i = 0; i < pages.size() && i < 8; ++i) { + const PageInfo& info = *pages[i].second; + pages_text += fmt::format( + "{}{:#x}: {} reads, {} waits in {:.1f} ms, lists at {:#x}-{:#x}, last written " + "{:#x}-{:#x} (of {} bytes) by {}", + pages_text.empty() ? "" : "; ", pages[i].first, info.waits.reads, info.waits.waits, + Ms(info.waits.wait_ns), info.list_begin, info.list_end, info.write_begin, + info.write_end, info.write_size, WriterText(info.writer)); + } + if (!pages_text.empty()) { + LOG_INFO(HW_GPU, "list stats pages: {}", pages_text); + } +} + +inline void LogVerify(const State& state, double seconds) { + const Period& counts = state.period; + LOG_INFO(HW_GPU, + "list verify {:.1f}s: {} lists read again ({} drained and flushed), {} differ in {} " + "words: {} {}, {} {}, {} {}, {} {}; since the start {} of {} differ", + seconds, counts.verified, counts.verified_drained, counts.differ, counts.differ_words, + difference_names[0], counts.differences[0], difference_names[1], counts.differences[1], + difference_names[2], counts.differences[2], difference_names[3], counts.differences[3], + state.differ_total, state.verified_total); +} + +/// The front: logs every 10 s. +inline void MaybeLog() { + State& state = Get(); + const auto now = std::chrono::steady_clock::now(); + std::scoped_lock lock{state.mutex}; + const double seconds = std::chrono::duration(now - state.last_log).count(); + if (seconds < 10.0) { + return; + } + state.last_log = now; + if (g_enabled) { + LogStats(state, seconds); + } + if (g_verify) { + LogVerify(state, seconds); + } + state.period = {}; + state.writers.clear(); + for (auto& entry : state.pages) { + entry.second.waits = {}; + entry.second.list_begin = ~u32{0}; + entry.second.list_end = 0; + } +} + +/// The front starts reading a list of `words` on GPU-written pages. +inline void BeginList(u32 words) { + MaybeLog(); + t_served = {}; + State& state = Get(); + std::scoped_lock lock{state.mutex}; + state.parts.clear(); + ++state.period.lists; + state.period.list_words += words; + ++state.period.list_sizes[Bucket(words, list_size_limits)]; +} + +/// The front read `size` bytes at `offset` of the command list page at `page_addr`. `result`: as +/// Nx2ReadListCopy's result tells (PageSynced, CopyReady or Unknown; NoteServed tells more). +/// `after_sync`: it first waited for the work before the last sync (a page draws write). +/// `copy_equal`: for a read from the copy, whether guest memory held the same bytes. `backlog` +/// and `unpublished`: the ring words the back had not run before the read, and of them those it +/// did not see. `channel`: the list's. +inline void NoteRead(u64 page_addr, u32 offset, u32 size, Served result, bool after_sync, + std::optional copy_equal, u64 backlog, u64 unpublished, s32 channel) { + const ServedDetail detail = t_served; + t_served = {}; + const Served served = result == Served::Unknown || !detail.valid ? result : detail.served; + const bool waited = served == Served::CopyWaited; + const u64 wait_ns = waited ? detail.waited_ns : 0; + const bool other_channels = waited && OtherChannelsQueued(channel); + State& state = Get(); + std::scoped_lock lock{state.mutex}; + state.parts.push_back(ListPart{page_addr, offset, size, served}); + if (!g_enabled) { + return; + } + Period& counts = state.period; + ++counts.served[static_cast(served)]; + counts.after_sync += after_sync ? 1 : 0; + PageInfo& page = state.pages[page_addr]; + page.waits.Add(waited, wait_ns); + page.list_begin = (std::min)(page.list_begin, offset); + page.list_end = (std::max)(page.list_end, offset + size); + counts.by_writer[static_cast(page.writer.writer)].Add(waited, wait_ns); + if (page.writer.writer != Writer::None) { + WriterStats& stats = state.writers[WriterKey(page.writer)]; + stats.writer = page.writer; + stats.waits.Add(waited, wait_ns); + } + if (copy_equal) { + ++counts.copies_compared; + counts.copies_equal += *copy_equal ? 1 : 0; + } + if (!waited) { + return; + } + counts.wait_ns += wait_ns; + counts.wait_max_ns = (std::max)(counts.wait_max_ns, wait_ns); + ++counts.wait_sizes[Bucket(wait_ns / 1000, wait_limits_us)]; + if (backlog != 0) { + ++counts.waits_backlog; + counts.backlog_words += backlog; + } + if (unpublished != 0) { + ++counts.waits_unpublished; + counts.unpublished_words += unpublished; + } + counts.waits_other_channels += other_channels ? 1 : 0; + if (page.copy_ns != 0 && page.copy_ns <= detail.wait_start_ns) { + ++counts.waits_recorded; + counts.since_recorded_ns += detail.wait_start_ns - page.copy_ns; + } +} + +/// EDEN_NX2_LIST_VERIFY: the front read the list at `address` as `parsed`, then as before as +/// `reference` (`reference_drained`: by draining and flushing) and, when those differ, drained +/// and flushed as `drained` (else empty). +inline void NoteVerify(u64 address, std::span parsed, std::span reference, + std::span drained, bool reference_drained) { + u64 differing = 0; + size_t first = parsed.size(); + for (size_t i = 0; i < parsed.size() && i < reference.size(); ++i) { + if (parsed[i] != reference[i]) { + ++differing; + first = (std::min)(first, i); + } + } + State& state = Get(); + std::scoped_lock lock{state.mutex}; + Period& counts = state.period; + ++counts.verified; + ++state.verified_total; + counts.verified_drained += reference_drained ? 1 : 0; + if (differing == 0) { + return; + } + Difference kind = Difference::Drained; + if (!reference_drained) { + if (std::ranges::equal(drained, parsed)) { + kind = Difference::OldStale; + } else if (std::ranges::equal(drained, reference)) { + kind = Difference::NewStale; + } else { + kind = Difference::AllDiffer; + } + } + ++counts.differ; + ++state.differ_total; + counts.differ_words += differing; + ++counts.differences[static_cast(kind)]; + if (state.differ_logged >= 32) { + return; + } + ++state.differ_logged; + std::string parts_text; + for (const ListPart& part : state.parts) { + const auto it = state.pages.find(part.page); + const PageInfo info = it != state.pages.end() ? it->second : PageInfo{}; + parts_text += fmt::format("{}{:#x}+{:#x} ({} bytes) {}, last written {:#x}-{:#x} by {}", + parts_text.empty() ? "" : "; ", part.page, part.offset, part.size, + served_names[static_cast(part.served)], info.write_begin, + info.write_end, WriterText(info.writer)); + } + LOG_WARNING(HW_GPU, + "list verify: list at {:#x} ({} words) differs from its read as before in {} " + "words, the first word {}: {:#010x}, as before {:#010x}{}: {}; its pages: {}", + address, parsed.size(), differing, first, parsed[first], reference[first], + drained.size() > first ? fmt::format(", drained {:#010x}", drained[first]) + : std::string{}, + difference_names[static_cast(kind)], parts_text); +} + +} // namespace Tegra::Nx2ListStats diff --git a/src/video_core/nx2_perf.h b/src/video_core/nx2_perf.h index 0fe16cfb8e..2b17a690c9 100644 --- a/src/video_core/nx2_perf.h +++ b/src/video_core/nx2_perf.h @@ -7,12 +7,24 @@ namespace VideoCommon::Nx2Perf { +[[nodiscard]] inline bool VerifySwitchSet() { + const char* value = std::getenv("EDEN_NX2_PERF_VERIFY"); + return value != nullptr && value[0] == '1'; +} + +/// EDEN_NX2_PERF_VERIFY=1. A plain flag: Maxwell3D::CallMethod checks it on every 3D write, +/// and a function's static is checked for its initialization on every call (MSVC reads a +/// thread-local for it). +inline bool g_verify = VerifySwitchSet(); + +/// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads +/// nx2_switches.txt, so a switch set only there takes effect from that point on. +inline void ReadSwitch() { + g_verify = VerifySwitchSet(); +} + inline bool Verify() { - static const bool verify = [] { - const char* value = std::getenv("EDEN_NX2_PERF_VERIFY"); - return value != nullptr && value[0] == '1'; - }(); - return verify; + return g_verify; } } // namespace VideoCommon::Nx2Perf diff --git a/src/video_core/nx2_profile.h b/src/video_core/nx2_profile.h index 4538491db8..1160f1a769 100644 --- a/src/video_core/nx2_profile.h +++ b/src/video_core/nx2_profile.h @@ -64,6 +64,22 @@ enum class Zone : u32 { WriterWait, ListSubmitWait, ListGpuWait, + AcquirePoll, ///< TEST: a suspended acquire's poll (CanResumeAcquire in ResumePending) + Resnapshot, ///< TEST: command lists copied again at a resume (DmaPusher::RefreshSnapshot) + // TEST: fine zones, EDEN_NX2_GPU_PROFILE=2 only (FineScope): they split the zones above. + CbBind, ///< Of method: constant buffer binds (Maxwell3D::ProcessCBBind) + CbUpload, ///< Of method: constant buffer words written to memory + QueryReport, ///< Of method: report semaphores (Maxwell3D::ProcessQueryGet) + CounterReset, ///< Of method: counter resets (Maxwell3D::ProcessCounterReset) + RenderEnable, ///< Of method: render enable conditions (Maxwell3D::ProcessQueryCondition) + Sink, ///< State writes held for the next executable method (ConsumeSinkImpl) + MacroEnd, ///< A front macro's end on the back (Maxwell3D::FinishFrontMacro) + DrawSetup, ///< Of draw: after the pipeline lookup, before Configure (cache locks) + DrawQueries, ///< Of draw: query and transform feedback state before recording + TickWork, ///< Of draw and compute: the sync operations after them (GPU::TickWork) + UniformBuffers, ///< Of bind stage buffers: the uniform buffers + GeometryBuffers, ///< Of update buffers: index and vertex buffer binds + StorageBindings, ///< Of read handles: storage buffer addresses from constant buffers GraphicsPipelineMiss, ///< A graphics pipeline built on the GPU thread ComputePipelineMiss, ///< A compute pipeline built on the GPU thread PipelineWait, ///< VulkanWorker: a draw or dispatch waits for its pipeline's build @@ -85,11 +101,16 @@ inline constexpr std::array(Zone::Count)> zone_ "descriptor acquire", "bind stage buffers", "push descriptors", "render targets", "configure draw", "dynamic state", "draw record", "texture upload", "fence", "fence skipped", "method", "split drain", "split wait", - "writer wait", "list submit wait", "list GPU wait", "graphics pipeline miss", + "writer wait", "list submit wait", "list GPU wait", "acquire poll", "resnapshot", + "cb bind", "cb upload", "query report", "counter reset", "render enable", "sink", + "macro end", "draw setup", "draw queries", "tick work", "uniform buffers", + "geometry buffers", "storage bindings", + "graphics pipeline miss", "compute pipeline miss", "pipeline wait", "shader capture", "texture cache", "GPU wait", "present", "worker idle", "present wait", "queue submit", }; +static_assert(zone_names.back() != nullptr, "a zone without a name"); static_assert(static_cast(Zone::Count) <= StutterTrace::ZoneSlots); @@ -133,10 +154,19 @@ inline bool g_enabled = std::getenv("EDEN_NX2_GPU_PROFILE") != nullptr; /// Whether zones measure: for the profile or for EDEN_NX2_STUTTER_TRACE. inline bool g_zones_enabled = g_enabled || StutterTrace::ReadEnv(); +/// TEST: EDEN_NX2_GPU_PROFILE=2 adds the fine zones (FineScope) and counters (FineCounter), at +/// the cost of more time stamp reads: =1 keeps the zones runs compare with. +[[nodiscard]] inline bool FineSwitchSet() { + const char* value = std::getenv("EDEN_NX2_GPU_PROFILE"); + return value != nullptr && value[0] == '2'; +} +inline bool g_fine = FineSwitchSet(); + /// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads /// nx2_switches.txt, so a switch set only there takes effect from that point on. inline void ReadSwitch() { g_enabled = std::getenv("EDEN_NX2_GPU_PROFILE") != nullptr; + g_fine = FineSwitchSet(); StutterTrace::ReadSwitch(); g_zones_enabled = g_enabled || StutterTrace::Enabled(); Common::g_count_spin_contention = g_enabled; @@ -146,6 +176,10 @@ inline bool Enabled() { return g_enabled; } +inline bool Fine() { + return g_fine; +} + // Nanoseconds; from the host's own counter where the wall clock reads it directly (the zones // run ~150,000 times a second, and QueryPerformanceCounter cost ~5% of the GPU thread). inline u64 Now() { @@ -163,10 +197,30 @@ inline u64 Ticks() { #endif } +/// TEST: amounts counted with EDEN_NX2_GPU_PROFILE=2, logged per second in a line of their own. +enum class FineCounter : u32 { + UniformRebinds, ///< Small uniform buffers bound again from the same stream allocation + UniformStreamed, ///< Small uniform buffers copied into the stream buffer + UniformCached, ///< Uniform buffers bound from their host buffer + UniformOther, ///< Uniform buffers bound otherwise (root tables, OpenGL, fallbacks) + UniformBytes, ///< Bytes of the uniform buffers bound (all paths) + StorageBinds, ///< Graphics storage buffers bound + TextureHandles, ///< Graphics texture handles read + Count, +}; + +inline constexpr std::array(FineCounter::Count)> + fine_counter_names{ + "uniform rebinds", "uniform streamed", "uniform cached", "uniform other", + "uniform bytes", "storage binds", "texture handles", + }; +static_assert(fine_counter_names.back() != nullptr, "a fine counter without a name"); + struct ThreadData { + std::array(FineCounter::Count)> fine_counters{}; ///< TEST std::array(Zone::Count)> exclusive_ns{}; std::array(Zone::Count)> count{}; - std::array counters{}; ///< See Counter + std::array counters{}; ///< See Counter std::array(DrainSite::Count)> drain_ns{}; std::array(DrainSite::Count)> drain_count{}; /// GPU waits by the zone they happened in (Other: outside every zone) @@ -177,18 +231,27 @@ struct ThreadData { u64 window_start{}; u64 window_start_ticks{}; std::optional cpu_start_ns; + u32 depth{}; ///< Zones open (Scope, timed SampledScope) + u32 samples_left{1}; ///< SampledScope: entries until the next timed one + u32 sample_weight{1}; ///< SampledScope: the entries the next timed one stands for + u32 sample_seed{0x2545F491}; }; inline thread_local ThreadData t_data; class Scope { public: - explicit Scope(Zone zone_) : zone{zone_}, enabled{g_zones_enabled} { + explicit Scope(Zone zone_) : Scope{zone_, g_zones_enabled} {} + + /// A zone timed only if `on` (a profiled run's choice, such as Fine()). + Scope(Zone zone_, bool on) : zone{zone_}, enabled{on} { if (!enabled) { return; } - parent_child_ns = t_data.child_ns; - t_data.child_ns = 0; + auto& data = t_data; + ++data.depth; + parent_child_ns = data.child_ns; + data.child_ns = 0; parent = t_data.current; t_data.current = zone; start = Ticks(); @@ -211,6 +274,7 @@ class Scope { data.exclusive_ns[index] += exclusive; ++data.count[index]; data.child_ns = parent_child_ns + elapsed; + --data.depth; data.current = parent; if (zone == Zone::GpuWait) { data.gpu_wait_ns[static_cast(parent)] += elapsed; @@ -237,6 +301,111 @@ class Scope { u64 parent_child_ns{}; }; +/// TEST: a fine zone (EDEN_NX2_GPU_PROFILE=2). +class FineScope : public Scope { +public: + explicit FineScope(Zone zone_) : Scope{zone_, Fine()} {} +}; + +/// The ticks two time stamp reads in a row take: what a timed entry measures on top of its own +/// time (the least of a few tries). +inline u64 TimingOverheadTicks() { + static const u64 overhead = [] { + u64 least = ~u64{0}; + for (int i = 0; i < 64; ++i) { + const u64 first = Ticks(); + const u64 second = Ticks(); + least = (std::min)(least, second - first); + } + return least; + }(); + return overhead; +} + +/// A zone entered millions of times a second, such as the split's back's 3D register writes +/// (Maxwell3D::CallMethod): timing every entry took two time stamp reads each, which cost more +/// than most entries and slowed the very runs it measured. Entered outside every other zone, one +/// entry in about SampleEvery is timed and stands for the entries since the last timed one; all +/// are counted. Entered inside another zone it does nothing: the entry is that zone's time. +class SampledScope { +public: + static constexpr u32 SampleEvery = 32; + /// About 25 us at 4 GHz (time stamp ticks): far longer than these entries take. + static constexpr u64 LongTicks = 100'000; + + /// `entries`: what this entry counts as (a run of constant buffer words counts each word). + explicit SampledScope(Zone zone, u32 entries = 1) { + if (!g_zones_enabled) { + return; + } + auto& data = t_data; + if (data.depth != 0) { + return; + } + index = static_cast(zone); + data.count[index] += entries; + if (--data.samples_left != 0) { + return; + } + weight = data.sample_weight; + // The next interval: 1 to 2 * SampleEvery - 1 entries, so that no fixed period lines up + // with a repeating pattern of entries (xorshift32). + u32 seed = data.sample_seed; + seed ^= seed << 13; + seed ^= seed >> 17; + seed ^= seed << 5; + data.sample_seed = seed; + data.sample_weight = 1 + seed % (2 * SampleEvery - 1); + data.samples_left = data.sample_weight; + ++data.depth; + parent_child_ns = data.child_ns; + data.child_ns = 0; + parent = data.current; + data.current = zone; + start = Ticks(); + traced = StutterTrace::Enabled(); + if (traced) { + outer = StutterTrace::Open(stutter_categories[index], start, index); + } + } + + ~SampledScope() { + if (weight == 0) { + return; + } + const u64 end = Ticks(); + const u64 elapsed = end - start; + auto& data = t_data; + // Less the reads' own cost, which the untimed entries did not pay. + const u64 overhead = TimingOverheadTicks(); + const u64 own = elapsed - data.child_ns; + const u64 measured = own > overhead ? own - overhead : 0; + // An entry this long was most likely interrupted or preempted: it counts once, not for + // the untimed entries too. + data.exclusive_ns[index] += measured > LongTicks ? measured : measured * weight; + data.child_ns = parent_child_ns + elapsed; + --data.depth; + data.current = parent; + if (traced) { + StutterTrace::Close(outer, end, + parent == Zone::Other ? StutterTrace::ZoneSlots + : static_cast(parent)); + } + } + + SampledScope(const SampledScope&) = delete; + SampledScope& operator=(const SampledScope&) = delete; + +private: + size_t index{}; + u64 weight{}; ///< 0: not timed + u64 start{}; + u64 parent_child_ns{}; + Zone parent{Zone::Other}; + bool traced{}; + StutterTrace::Category outer{StutterTrace::Category::None}; +}; + /// Amounts counted on the GPU thread, logged per second with the zones. enum class Counter : u32 { DownloadBytes, ///< Buffer data copied back from the host GPU at fences @@ -247,6 +416,12 @@ enum class Counter : u32 { SameValueWrites, ///< Of MethodWrites: the register already held the value AcquireChecks, ///< Semaphore acquires evaluated AcquireWaits, ///< Of AcquireChecks: not met yet (the channel waits) + /// TEST: the split's front publishes words while its back waits for work (GpuSplit::Publish): + /// the back's wait since it ran out or since the first of them was sent (Ticks), and count + LatePublishTicks, + LatePublishes, + ResnapshotWords, ///< TEST: command list words copied again at resumes + EarlyPublishes, ///< TEST: publishes to a back waiting for work, before publish_every }; /// Method calls on this thread by method (3D registers on the back, macros on the front), and @@ -263,6 +438,18 @@ inline void CountMethod(u32 method, bool same) { } } +/// A 3D register write (Maxwell3D::CallMethod below the macro range), `same` if the register +/// held the value already: the write counters and the per-method counts. +inline void CountRegisterWrite(u32 method, bool same) { + if (!Enabled()) { + return; + } + auto& counters = t_data.counters; + ++counters[static_cast(Counter::MethodWrites)]; + counters[static_cast(Counter::SameValueWrites)] += same ? 1 : 0; + CountMethod(method, same); +} + inline void CountDmaRead(u32 macro) { if (Enabled() && macro < t_dma_reads.size()) { ++t_dma_reads[macro]; @@ -307,6 +494,13 @@ inline void Add(Counter counter, u64 amount) { } } +/// TEST: EDEN_NX2_GPU_PROFILE=2 only. +inline void AddFine(FineCounter counter, u64 amount) { + if (Fine()) [[unlikely]] { + t_data.fine_counters[static_cast(counter)] += amount; + } +} + /// The front waited for the back since `start` (Ticks) for `site`. inline void AddDrain(DrainSite site, u64 start) { if (Enabled()) { @@ -367,6 +561,15 @@ inline void Report(const char* thread_name = "GPU thread", bool report_all_conte line += fmt::format(" | acquires {:.0f} ({:.0f} waited)", static_cast(data.counters[6]) * 1e9 / window, static_cast(data.counters[7]) * 1e9 / window); + if (data.counters[9] != 0 || data.counters[10] != 0 || data.counters[11] != 0) { + // TEST: the back idle while words it could run waited on the front. + line += fmt::format(" | publishes to an idle back late by {:.1f} ms ({:.0f}/s), early " + "{:.0f}/s | resnapshot words {:.0f}/s", + static_cast(data.counters[8]) * 1000.0 / window_ticks, + static_cast(data.counters[9]) * 1e9 / window, + static_cast(data.counters[11]) * 1e9 / window, + static_cast(data.counters[10]) * 1e9 / window); + } std::string drains; for (size_t i = 0; i < data.drain_count.size(); ++i) { if (data.drain_count[i] != 0) { @@ -418,6 +621,20 @@ inline void Report(const char* thread_name = "GPU thread", bool report_all_conte static_cast(window) / 1e6); } LOG_INFO(HW_GPU, "{} per second:{}", thread_name, line); + if (Fine()) { + // TEST: the fine counters. + std::string fine; + for (size_t i = 0; i < data.fine_counters.size(); ++i) { + if (data.fine_counters[i] != 0) { + fine += fmt::format(" {} {:.0f},", fine_counter_names[i], + static_cast(data.fine_counters[i]) * 1e9 / window); + } + } + if (!fine.empty()) { + fine.pop_back(); + LOG_INFO(HW_GPU, "{} fine counters per second:{}", thread_name, fine); + } + } if (const std::string methods = TopMethods(t_method_count, &t_method_same, window); !methods.empty()) { LOG_INFO(HW_GPU, "{} methods per second:{}", thread_name, methods); } @@ -427,8 +644,10 @@ inline void Report(const char* thread_name = "GPU thread", bool report_all_conte t_method_count = {}; t_method_same = {}; t_dma_reads = {}; + const u32 depth = data.depth; const Zone current = data.current; data = {}; + data.depth = depth; data.current = current; data.window_start = now; data.window_start_ticks = now_ticks; diff --git a/src/video_core/nx2_read_trace.h b/src/video_core/nx2_read_trace.h new file mode 100644 index 0000000000..7c798e802f --- /dev/null +++ b/src/video_core/nx2_read_trace.h @@ -0,0 +1,535 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "common/common_types.h" +#include "common/logging.h" +#include "video_core/gpu_wait_stats.h" + +// TEST (EDEN_NX2_READ_TRACE=1): where a guest read of GPU-written memory spends its wait. +// +// Each read that waits (GPU::OnCPURead) is placed on the host GPU's timeline. Vulkan writes a +// device timestamp at the start and end of every submission, after each draw or dispatch that +// writes pages the guest reads back (the area's last such write before the read's copy is "the +// write"), and before and after the read's copy. The reading thread reads the device's clock +// (VK_EXT_calibrated_timestamps) once the flush is asked for, which places the request on that +// timeline. The wait then splits into: until the write's work ended on the GPU (the GPU still +// had it, or work queued before it, to run), until the copy could start (work queued after the +// write), the copy, and the reading thread's wake-up, lock and copy into guest memory. Also kept: +// where the write was at the request (done, on the GPU, not yet queued) and how long before it +// the GPU thread recorded it, the work recorded between the write and the copy, the submissions +// in flight at the request with the GPU time they still ran, and how long the copy waited for +// its vkQueueSubmit. In the frames EDEN_DKB_FRAME_TRACE samples (each draw and dispatch timed), +// the GPU time before the copy also splits into timed draws and dispatches and the rest. +// +// Every 10 s: one "Read trace" line for all reads, and one for each of the areas that waited +// longest. Times are per read, in ms. +namespace VideoCommon::ReadTrace { + +[[nodiscard]] inline bool SwitchSet() { + const char* value = std::getenv("EDEN_NX2_READ_TRACE"); + return value != nullptr && *value != '\0' && *value != '0'; +} + +// A plain flag: checked at every read download and fence. +inline bool g_enabled = SwitchSet(); + +/// Called by VideoCore::CreateGPU: static initialization runs before the frontend loads +/// nx2_switches.txt, so a switch set only there takes effect from that point on. +inline void ReadSwitch() { + g_enabled = SwitchSet(); +} + +[[nodiscard]] inline bool Enabled() { + return g_enabled; +} + +/// Nanoseconds, the clock of the GPU wait stats (the same on every thread). +[[nodiscard]] inline u64 Now() { + return GpuWaitStats::Now(); +} + +/// Work the GPU thread recorded so far. +struct Counts { + u64 draws{}; + u64 dispatches{}; + u64 render_passes{}; +}; + +/// The area's last GPU write before the read's copy was recorded. +struct Write { + u64 tick{}; ///< The submission it went in + u64 recorded{}; ///< Now() once the writing draw or dispatch was recorded + u64 mark{}; ///< Device timestamp after it (Vulkan::Scheduler::TraceMark), 0 if none + Counts counts{}; +}; + +/// The copy a read waited for (BufferCache::DownloadMemoryForRead). +struct Copy { + u64 tick{}; ///< Its submission + u64 flushed{}; ///< Now() once the GPU thread had handed it over for submission + u64 start{}; ///< Device timestamp before the copies: all earlier work done + u64 end{}; ///< After them + u64 sequence{}; ///< Writes the buffer cache noted before it have a lower one + Counts counts{}; + bool early{}; ///< Recorded right behind its write (BufferCache::RecordEarlyDownloads) +}; + +enum class Step : u8 { + Request, ///< As the reading thread asks for the flush + Calibrate, ///< While it waits for the GPU thread to run it: the device's clock + Done, ///< The bytes are in guest memory +}; + +/// One guest read that waited for the GPU, filled in as it goes. +struct Read { + u64 area{}; + bool in_flight{}; ///< Waited for an earlier read's copy + // Step::Request + u64 request{}; + u64 completed{}; ///< The last submission the host GPU completed + u64 submitted{}; ///< The last one vkQueueSubmit returned for + u64 recording{}; ///< The one being recorded: those before it went to the Vulkan worker + // Step::Calibrate: the device's timestamp `device` at Now() `host` (+- `uncertainty`) + u64 device{}; + u64 host{}; + u64 uncertainty{}; + // FinishReadDownloads (the reading thread) + bool has_copy{}; + Copy copy{}; + bool has_write{}; + Write write{}; + u64 woke{}; ///< The copy's submission was complete + u64 locked{}; ///< Then the buffer cache lock was taken + u64 landed{}; ///< Then the bytes were in guest memory +}; + +/// The reading thread's read while it is in RasterizerInterface::FinishRead. +inline thread_local Read* t_read{}; + +/// Where a read's wait went on the device's timeline (Vulkan::Scheduler, Step::Done). +struct Timeline { + bool placed{}; ///< Calibrated, and the timestamps it needs were read + /// ns from the request: until the write ended, until the copy could start, the copy, until + /// the reading thread woke. They add up to `woke - request`. + u64 to_write{}; + u64 to_copy{}; + u64 copy{}; + u64 to_wake{}; + /// Of to_write and to_copy, the time the host GPU ran no submission + u64 idle_to_write{}; + u64 idle_to_copy{}; + /// GPU time the submissions in flight at the request still ran after it + u64 backlog_busy{}; + /// The write's work ended this long before the request (0: after it) + u64 write_lead{}; + /// From the write's end to the copy's start on the GPU, wherever the request fell + u64 write_gap{}; + /// The copy's vkQueueSubmit returned this long after the GPU thread handed it over + bool has_submit{}; + u64 submit{}; + /// Device ticks: the request, the write's end (not before the request), the copy's start + std::array window{}; +}; + +/// The device timestamps placing a read takes (Vulkan::Scheduler). +struct DeviceTimes { + double ns_per_tick{}; + u64 write_end{}; + u64 copy_start{}; + u64 copy_end{}; + /// Start and end of each submission after the last one the host GPU had completed at the + /// request (read.completed + 1 on), up to the copy's. + std::span> submissions; +}; + +/// Splits a read's wait on the device's timeline. +[[nodiscard]] inline Timeline Place(const Read& read, const DeviceTimes& times) { + // ns from the request: the device read `device` at Now() `host`. + const auto at = [&](u64 ticks) { + return static_cast(static_cast(ticks - read.device)) * times.ns_per_tick + + static_cast(read.host - read.request); + }; + // The host GPU's time running submissions in [lower, upper), those up to last_tick. + const auto busy = [&](double lower, double upper, u64 last_tick) { + double sum = 0.0; + double covered = lower; + for (size_t i = 0; i < times.submissions.size(); ++i) { + if (read.completed + 1 + i > last_tick) { + break; + } + const double from = (std::max)(at(times.submissions[i].first), covered); + const double to = (std::min)(at(times.submissions[i].second), upper); + if (to > from) { + sum += to - from; + covered = to; + } + } + return sum; + }; + const auto ns = [](double value) { return static_cast((std::max)(value, 0.0)); }; + const double woke = static_cast(read.woke - read.request); + const double write_ended = at(times.write_end); + const double written = std::clamp(write_ended, 0.0, woke); + const double copy_started = std::clamp(at(times.copy_start), written, woke); + const double copied = std::clamp(at(times.copy_end), copy_started, woke); + Timeline timeline{}; + timeline.placed = true; + timeline.to_write = ns(written); + timeline.to_copy = ns(copy_started - written); + timeline.copy = ns(copied - copy_started); + timeline.to_wake = ns(woke - copied); + timeline.idle_to_write = ns(written - busy(0.0, written, read.copy.tick)); + timeline.idle_to_copy = + ns(copy_started - written - busy(written, copy_started, read.copy.tick)); + timeline.backlog_busy = ns(busy(0.0, std::numeric_limits::infinity(), read.submitted)); + timeline.write_lead = ns(-write_ended); + timeline.write_gap = ns(at(times.copy_start) - write_ended); + const double request_ticks = static_cast(read.device) - + static_cast(read.host - read.request) / times.ns_per_tick; + const auto ticks_at = [&](double offset) { + return static_cast(request_ticks + offset / times.ns_per_tick); + }; + timeline.window = {ticks_at(0.0), ticks_at(written), ticks_at(copy_started)}; + return timeline; +} + +/// Early read downloads (BufferCache::RecordEarlyDownloads), any thread. +struct EarlyTotals { + std::atomic recorded{}; + std::atomic bytes{}; + std::atomic served_landed{}; ///< Reads that found their bytes in guest memory + std::atomic served_at_read{}; ///< Done on the host GPU: the read's lookup landed them + std::atomic served_in_flight{}; ///< Not yet done: the read waited for them + std::atomic gpu_side_waits{}; ///< Writes from the GPU's side that waited for one + std::atomic gpu_side_wait_ns{}; +}; +inline EarlyTotals g_early; + +inline void Count(std::atomic& counter, u64 amount = 1) { + counter.fetch_add(amount, std::memory_order_relaxed); +} + +struct Totals { + u64 reads{}; + u64 in_flight{}; + u64 early_copies{}; ///< The copy waited for was an early download + u64 no_copy{}; ///< Found no copy to wait for + u64 no_write{}; ///< No GPU write over the area seen since it was first read back + u64 copy_done{}; ///< The copy's submission was complete at the request + u64 wait{}; ///< Request until the bytes were in guest memory, all reads + u64 lock{}; ///< Reads with a copy: woke until locked + u64 land{}; ///< Locked until in guest memory + // Placed reads + u64 placed{}; + u64 to_write{}; + u64 to_copy{}; + u64 copy{}; + u64 to_wake{}; + u64 idle_to_write{}; + u64 idle_to_copy{}; + u64 backlog_busy{}; + u64 uncertainty{}; + u64 write_ended{}; ///< The write's work had ended at the request (write_lead is over these) + u64 write_lead{}; + u64 write_gap{}; + // Reads with a copy: the submissions in flight, and those not yet queued, at the request + u64 backlog_subs{}; + u64 unqueued_subs{}; + u64 submits{}; + u64 submit{}; + // Reads with a write: where it was at the request, when it was recorded, what came after it + u64 write_done{}; + u64 write_on_gpu{}; + u64 write_unqueued{}; ///< Flushed to the Vulkan worker, no vkQueueSubmit yet + u64 write_recording{}; ///< In the submission still being recorded + u64 written_before{}; ///< Recorded before the request (since_write is over these) + u64 since_write{}; + u64 draws{}; + u64 dispatches{}; + u64 render_passes{}; + u64 submissions{}; + // Placed reads in the frames EDEN_DKB_FRAME_TRACE samples + u64 sampled{}; + u64 timed_to_write{}; + u64 busy_to_write{}; + u64 timed_to_copy{}; + u64 busy_to_copy{}; +}; + +/// A placed read, until its frame's timed work arrives (NoteTimedWork). +struct Window { + u64 area{}; + std::array ticks{}; + u64 busy_to_write{}; + u64 busy_to_copy{}; + bool pending{}; +}; + +struct State { + std::mutex mutex; + u64 window_start{}; + u64 presents{}; + Totals all; + std::vector> areas; + Totals other_areas; ///< Past MaxAreas + std::array windows{}; + size_t next_window{}; +}; +inline State g_state; +inline constexpr size_t MaxAreas = 64; +inline constexpr size_t ShownAreas = 4; + +/// `mutex` held. +inline Totals& AreaTotals(u64 area) { + for (auto& [start, totals] : g_state.areas) { + if (start == area) { + return totals; + } + } + if (g_state.areas.size() >= MaxAreas) { + return g_state.other_areas; + } + return g_state.areas.emplace_back(area, Totals{}).second; +} + +inline void Add(Totals& totals, const Read& read, const Timeline& timeline) { + ++totals.reads; + totals.in_flight += read.in_flight ? 1 : 0; + totals.wait += read.landed - read.request; + if (!read.has_copy) { + ++totals.no_copy; + return; + } + totals.copy_done += read.copy.tick <= read.completed ? 1 : 0; + totals.early_copies += read.copy.early ? 1 : 0; + totals.lock += read.locked - read.woke; + totals.land += read.landed - read.locked; + totals.backlog_subs += read.submitted > read.completed ? read.submitted - read.completed : 0; + totals.unqueued_subs += + read.recording > read.submitted + 1 ? read.recording - read.submitted - 1 : 0; + if (timeline.has_submit) { + ++totals.submits; + totals.submit += timeline.submit; + } + if (timeline.placed) { + ++totals.placed; + totals.to_write += timeline.to_write; + totals.to_copy += timeline.to_copy; + totals.copy += timeline.copy; + totals.to_wake += timeline.to_wake; + totals.idle_to_write += timeline.idle_to_write; + totals.idle_to_copy += timeline.idle_to_copy; + totals.backlog_busy += timeline.backlog_busy; + totals.uncertainty += read.uncertainty; + totals.write_ended += timeline.write_lead != 0 ? 1 : 0; + totals.write_lead += timeline.write_lead; + totals.write_gap += timeline.write_gap; + } + if (!read.has_write) { + ++totals.no_write; + return; + } + const Write& write = read.write; + if (write.tick <= read.completed) { + ++totals.write_done; + } else if (write.tick <= read.submitted) { + ++totals.write_on_gpu; + } else if (write.tick < read.recording) { + ++totals.write_unqueued; + } else { + ++totals.write_recording; + } + if (write.recorded <= read.request) { + ++totals.written_before; + totals.since_write += read.request - write.recorded; + } + const Counts& after = read.copy.counts; + totals.draws += after.draws - write.counts.draws; + totals.dispatches += after.dispatches - write.counts.dispatches; + totals.render_passes += after.render_passes - write.counts.render_passes; + totals.submissions += read.copy.tick - write.tick; +} + +inline std::string Format(u64 area, const Totals& t) { + const auto ms = [](u64 sum, u64 count) { + return count == 0 ? 0.0 : static_cast(sum) / 1e6 / static_cast(count); + }; + const auto per = [](u64 sum, u64 count) { + return count == 0 ? 0.0 : static_cast(sum) / static_cast(count); + }; + std::string line = area == 0 ? std::string{"all"} : fmt::format("{:#x}", area); + const u64 copied = t.reads - t.no_copy; + line += fmt::format(": {} reads ({} in flight, {} waited for an early download, {} without a " + "copy, {} with it done at the request, {} placed on the GPU timeline " + "+-{:.3f}), wait {:.2f}", + t.reads, t.in_flight, t.early_copies, t.no_copy, t.copy_done, t.placed, + ms(t.uncertainty, t.placed) / 2, ms(t.wait, t.reads)); + if (t.placed != 0) { + line += fmt::format(" | from the request: to the write's end {:.2f} (GPU idle {:.2f}), to " + "the copy's start {:.2f} (idle {:.2f}), copy {:.2f}, to the wake-up " + "{:.2f}", + ms(t.to_write, t.placed), ms(t.idle_to_write, t.placed), + ms(t.to_copy, t.placed), ms(t.idle_to_copy, t.placed), + ms(t.copy, t.placed), ms(t.to_wake, t.placed)); + } + if (copied != 0) { + line += fmt::format("{}lock {:.2f}, to guest memory {:.2f}", t.placed != 0 ? ", " : " | ", + ms(t.lock, copied), ms(t.land, copied)); + } + if (t.placed != 0) { + line += + fmt::format("; the write's end to the copy's start {:.2f}; the write had ended " + "at the request {}x, {:.2f} before it", + ms(t.write_gap, t.placed), t.write_ended, ms(t.write_lead, t.write_ended)); + } + if (copied != 0) { + line += fmt::format(" | at the request: {:.1f} submissions in flight", + per(t.backlog_subs, copied)); + if (t.placed != 0) { + line += fmt::format(" (GPU busy {:.2f} more)", ms(t.backlog_busy, t.placed)); + } + line += fmt::format(", {:.1f} flushed but not queued; the copy queued {:.3f} after its " + "flush", + per(t.unqueued_subs, copied), ms(t.submit, t.submits)); + } + const u64 written = copied - t.no_write; + if (written != 0) { + line += fmt::format(" | the write at the request: done {}, on the GPU {}, flushed but not " + "queued {}, still recording {}; recorded {:.2f} before it ({} after); " + "then {:.1f} draws, {:.1f} dispatches, {:.1f} render passes, {:.1f} " + "submissions until the copy", + t.write_done, t.write_on_gpu, t.write_unqueued, t.write_recording, + ms(t.since_write, t.written_before), written - t.written_before, + per(t.draws, written), per(t.dispatches, written), + per(t.render_passes, written), per(t.submissions, written)); + } + if (t.no_write != 0) { + line += fmt::format(" ({} without a write seen)", t.no_write); + } + if (t.sampled != 0) { + line += + fmt::format(" | sampled frames, {} reads: draws and dispatches {:.2f} of the {:.2f} " + "busy to the write's end, {:.2f} of {:.2f} to the copy's start", + t.sampled, ms(t.timed_to_write, t.sampled), ms(t.busy_to_write, t.sampled), + ms(t.timed_to_copy, t.sampled), ms(t.busy_to_copy, t.sampled)); + } + return line; +} + +/// `mutex` held. +inline void MaybeReport(u64 now) { + const u64 presents = GpuWaitStats::g_presents.load(std::memory_order_relaxed); + if (g_state.window_start == 0) { + g_state.window_start = now; + g_state.presents = presents; + return; + } + if (now - g_state.window_start < GpuWaitStats::WindowNs) { + return; + } + std::ranges::sort(g_state.areas, std::greater<>{}, + [](const auto& entry) { return entry.second.wait; }); + LOG_INFO(HW_GPU, "Read trace (EDEN_NX2_READ_TRACE) in {:.1f} s, {} frames: {}", + static_cast(now - g_state.window_start) / 1e9, presents - g_state.presents, + Format(0, g_state.all)); + const auto take = [](std::atomic& counter) { + return counter.exchange(0, std::memory_order_relaxed); + }; + const u64 gpu_side_waits = take(g_early.gpu_side_waits); + LOG_INFO(HW_GPU, + "Read trace early downloads: {} recorded ({} KiB) | reads they served: {} found " + "the bytes in guest memory, {} found them done on the host GPU (landed at the read), " + "{} waited for them (in the lines above) | writes from the GPU's side that waited " + "for one: {} ({:.2f} ms)", + take(g_early.recorded), take(g_early.bytes) / 1024, take(g_early.served_landed), + take(g_early.served_at_read), take(g_early.served_in_flight), gpu_side_waits, + static_cast(take(g_early.gpu_side_wait_ns)) / 1e6); + for (size_t i = 0; i < g_state.areas.size() && i < ShownAreas; ++i) { + LOG_INFO(HW_GPU, "Read trace area {}", + Format(g_state.areas[i].first, g_state.areas[i].second)); + } + g_state.window_start = now; + g_state.presents = presents; + g_state.all = {}; + g_state.areas.clear(); + g_state.other_areas = {}; +} + +/// The reading thread, once its read is done (from Vulkan::Scheduler). +inline void Record(const Read& read, const Timeline& timeline) { + std::scoped_lock lock{g_state.mutex}; + Add(g_state.all, read, timeline); + Add(AreaTotals(read.area), read, timeline); + if (timeline.placed) { + g_state.windows[g_state.next_window++ % g_state.windows.size()] = Window{ + .area = read.area, + .ticks = timeline.window, + .busy_to_write = timeline.to_write - timeline.idle_to_write, + .busy_to_copy = timeline.to_copy - timeline.idle_to_copy, + .pending = true, + }; + } + MaybeReport(Now()); +} + +/// Once a frame (BufferCache::TickFrame): reports even while no read waits. +inline void Tick() { + if (!Enabled()) { + return; + } + std::scoped_lock lock{g_state.mutex}; + MaybeReport(Now()); +} + +/// The Vulkan worker, with the draws and dispatches of a frame EDEN_DKB_FRAME_TRACE timed +/// ([begin, end] device ticks, in order): splits the GPU time of the reads placed in it. +inline void NoteTimedWork(std::span> work, double ns_per_tick) { + if (work.empty()) { + return; + } + const u64 first = work.front().first; + const u64 last = work.back().second; + const auto timed = [&](u64 lower, u64 upper) { + u64 ticks = 0; + for (const auto& [begin, end] : work) { + const u64 from = (std::max)(begin, lower); + const u64 to = (std::min)(end, upper); + ticks += to > from ? to - from : 0; + } + return static_cast(static_cast(ticks) * ns_per_tick); + }; + std::scoped_lock lock{g_state.mutex}; + for (Window& window : g_state.windows) { + if (!window.pending || window.ticks[0] < first || window.ticks[2] > last) { + continue; + } + window.pending = false; + const u64 timed_to_write = timed(window.ticks[0], window.ticks[1]); + const u64 timed_to_copy = timed(window.ticks[1], window.ticks[2]); + for (Totals* totals : {&g_state.all, &AreaTotals(window.area)}) { + ++totals->sampled; + totals->timed_to_write += timed_to_write; + totals->busy_to_write += window.busy_to_write; + totals->timed_to_copy += timed_to_copy; + totals->busy_to_copy += window.busy_to_copy; + } + } +} + +} // namespace VideoCommon::ReadTrace diff --git a/src/video_core/query_cache/query_cache.h b/src/video_core/query_cache/query_cache.h index 7a971b3666..8889a80396 100644 --- a/src/video_core/query_cache/query_cache.h +++ b/src/video_core/query_cache/query_cache.h @@ -21,6 +21,7 @@ #include "common/settings.h" #include "video_core/engines/maxwell_3d.h" #include "video_core/gpu.h" +#include "video_core/gpu_split.h" #include "video_core/host1x/gpu_device_memory_manager.h" #include "video_core/memory_manager.h" #include "video_core/nx2.h" @@ -283,20 +284,37 @@ void QueryCacheBase::CounterReport(GPUVAddr addr, QueryType counter_type }; u8* pointer = impl->device_memory.template GetPointer(cpu_addr); u8* pointer_timestamp = impl->device_memory.template GetPointer(cpu_addr + 8); + // A fence-flagged payload release writes a constant, so at low accuracy with Default fences + // it stays synchronized as before #337: its fence may be skipped and the query freed at once. + // Only reports of GPU counters wait for their fence to run. + const bool is_synced = is_fence && counter_type == QueryType::Payload && + (Settings::IsGPUFenceBehaviorDefault() + ? !Settings::IsGPULevelHigh() + : !Settings::IsGPUFenceBehaviorBalanced() && + !Settings::IsGPUFenceBehaviorAccurate()); + // The GPU thread split's back entry this report came in: the front counts the report in + // memory once its operation ran with a fence (GpuSplit::NoteFencedWrite). A synced release's + // operation doesn't run in fence order, so it must not count earlier reports as landed. + const std::optional split_entry = + is_synced ? std::nullopt : VideoCommon::GpuSplit::Get().BackEntry(); const u64 report_size = has_timestamp ? 16 : 4; const bool immediate = !is_fence && !Settings::IsGPULevelHigh() && counter_type == QueryType::Payload; - // Written later: the buffer cache keeps GPU writes over it until then (NoteReportPending). + // Synced payload callbacks can run ahead of older fenced reports at the same address. + // They must not retire one of those reports from the buffer cache's ordered tracking. const bool report_noted = - !immediate && impl->rasterizer.NoteReportPending(cpu_addr, report_size); - std::function operation([this, streamer, query_base = query, query_location, - pointer, pointer_timestamp, cpu_addr, addr, - asid = gpu_memory->GetID(), counter_type, mirrored, + !immediate && !is_synced && impl->rasterizer.NoteReportPending(cpu_addr, report_size); + std::function operation([this, is_synced, streamer, query_base = query, + query_location, pointer, pointer_timestamp, cpu_addr, addr, + asid = gpu_memory->GetID(), counter_type, split_entry, mirrored, report_size, report_noted] { SCOPE_EXIT { if (report_noted) { impl->rasterizer.FinishReport(cpu_addr, report_size); } + if (split_entry) { + VideoCommon::GpuSplit::Get().NoteFencedWrite(*split_entry); + } }; if (True(query_base->flags & QueryFlagBits::IsInvalidated)) { if (const u64 trace = QueryTrace::Sample(QueryTrace::Event::Discard)) { @@ -306,7 +324,9 @@ void QueryCacheBase::CounterReport(GPUVAddr addr, QueryType counter_type trace, asid, addr, cpu_addr, static_cast(counter_type), static_cast(query_base->flags)); } - impl->pending_unregister.push_back(query_location); + if (!is_synced) { + impl->pending_unregister.push_back(query_location); + } return; } if (False(query_base->flags & QueryFlagBits::IsFinalValueSynced)) [[unlikely]] { @@ -319,7 +339,9 @@ void QueryCacheBase::CounterReport(GPUVAddr addr, QueryType counter_type } LOG_ERROR(HW_GPU, "Query report value not synchronized. Consider increasing GPU accuracy."); - impl->pending_unregister.push_back(query_location); + if (!is_synced) { + impl->pending_unregister.push_back(query_location); + } return; } query_base->value += streamer->GetAmendValue(); @@ -341,7 +363,10 @@ void QueryCacheBase::CounterReport(GPUVAddr addr, QueryType counter_type if (!mirrored && counter_type == QueryType::Payload) { shadow.NoteDirectRelease(cpu_addr, addr); } - Tegra::Nx2SemaphoreWaiters::Get().OnWrite(cpu_addr, static_cast(query_base->value)); + Tegra::Nx2SemaphoreWaiters::Get().OnWrite( + cpu_addr, static_cast(query_base->value), + counter_type == QueryType::Payload ? Tegra::Nx2SemaphoreWaiters::Source::PayloadRelease + : Tegra::Nx2SemaphoreWaiters::Source::QueryResult); if (const u64 trace = QueryTrace::Sample(QueryTrace::Event::Write)) { const bool wide = True(query_base->flags & QueryFlagBits::HasTimestamp); LOG_INFO(HW_GPU, @@ -351,12 +376,18 @@ void QueryCacheBase::CounterReport(GPUVAddr addr, QueryType counter_type wide ? query_base->value : static_cast(query_base->value), static_cast(query_base->flags)); } - impl->pending_unregister.push_back(query_location); + if (!is_synced) { + impl->pending_unregister.push_back(query_location); + } }); if (is_fence) { - VideoCore::g_signaling_query_report_fence = true; + VideoCore::g_signaling_query_report_fence = !is_synced; impl->rasterizer.SignalFence(std::move(operation)); VideoCore::g_signaling_query_report_fence = false; + if (is_synced) { + streamer->Free(new_query_id); + return; + } } else { if (immediate) { FinishReads(cpu_addr, has_timestamp ? 16 : 4); @@ -376,7 +407,8 @@ void QueryCacheBase::CounterReport(GPUVAddr addr, QueryType counter_type if (!mirrored) { shadow.NoteDirectRelease(cpu_addr, addr); } - Tegra::Nx2SemaphoreWaiters::Get().OnWrite(cpu_addr, payload); + Tegra::Nx2SemaphoreWaiters::Get().OnWrite( + cpu_addr, payload, Tegra::Nx2SemaphoreWaiters::Source::PayloadRelease); if (const u64 trace = QueryTrace::Sample(QueryTrace::Event::Write)) { LOG_INFO(HW_GPU, "DKB_QUERY write n={} path=immediate as={} gpu={:#x} device={:#x} type={} " @@ -684,7 +716,7 @@ template void QueryCacheBase::FinishReads(DAddr address, size_t size) { // SemiFlushQueryDirty asks with cache_mutex held, which no holder of the buffer cache's lock // takes. - if (VideoCore::ReadDownloadsMayCover(address, size)) { + if (VideoCore::ReadDownloadsMayCover(address, size)) [[unlikely]] { if (VideoCore::ReadCheckWrites::Enabled()) { VideoCore::ReadCheckWrites::Note(address, size, "a query result"); } diff --git a/src/video_core/rasterizer_download_area.h b/src/video_core/rasterizer_download_area.h index 27d5512e38..e2e2ad26bb 100644 --- a/src/video_core/rasterizer_download_area.h +++ b/src/video_core/rasterizer_download_area.h @@ -6,19 +6,27 @@ #include #include #include +#include #include #include #include #include "common/common_types.h" -#include "video_core/nx2_async_readback.h" namespace VideoCore { +/// The cache that reported a flush area (GetFlushArea), for statistics (GpuWaitStats). +enum class DownloadAreaSource : u8 { + None, + TextureCache, + BufferCache, +}; + struct RasterizerDownloadArea { VAddr start_address; VAddr end_address; bool preemtive; + DownloadAreaSource source{}; /// What guest memory lacks there is on its way for an earlier read: nothing to flush, only to /// wait for (RasterizerInterface::FinishRead). bool in_flight{}; @@ -29,8 +37,6 @@ struct RasterizerDownloadArea { /// the GPU's side first finishes those it overlaps (FinishRead), so none lands over it later. inline std::atomic g_read_downloads{}; -/// Tegra::Nx2MayBeGpuWritten on Switch 2 titles, true otherwise. Defined in gpu.cpp: nx2.h can't -/// be included here, since the Qt frontend reaches this header and its `slots` macro breaks it. [[nodiscard]] bool MayHoldGpuWrites(DAddr addr, u64 size); /// Whether such a download may still land over [addr, addr + size): what a write to guest memory @@ -39,7 +45,8 @@ inline std::atomic g_read_downloads{}; /// (Tegra::Nx2MayBeGpuWritten): writes to other pages skip the lock, which the GPU thread holds /// for nearly every command. [[nodiscard]] inline bool ReadDownloadsMayCover(DAddr addr, u64 size) { - return g_read_downloads.load(std::memory_order_acquire) != 0 && MayHoldGpuWrites(addr, size); + return g_read_downloads.load(std::memory_order_acquire) != 0 && + MayHoldGpuWrites(addr, size); } /// What DownloadLanding publishes. @@ -49,6 +56,7 @@ struct DownloadLandingState { std::atomic begin{}; std::atomic end{}; std::atomic waits{}; + std::atomic wait_ns{}; ///< TEST (EDEN_NX2_READ_CHECK) }; /// The buffer cache writing downloaded GPU bytes to guest memory [begin, end). A download keeps the @@ -82,7 +90,7 @@ class DownloadLanding { static void Await(DAddr addr, u64 size) noexcept { std::atomic_thread_fence(std::memory_order_seq_cst); const u64 sequence = state.sequence.load(std::memory_order_acquire); - if ((sequence & 1) != 0) { + if ((sequence & 1) != 0) [[unlikely]] { Wait(sequence, addr, size); } } @@ -92,6 +100,11 @@ class DownloadLanding { return state.waits.load(std::memory_order_relaxed); } + /// TEST (EDEN_NX2_READ_CHECK): how long they waited, in total. + [[nodiscard]] static u64 WaitNs() noexcept { + return state.wait_ns.load(std::memory_order_relaxed); + } + private: static void Wait(u64 sequence, DAddr addr, u64 size) noexcept { const DAddr begin = state.begin.load(std::memory_order_relaxed); @@ -104,21 +117,28 @@ class DownloadLanding { return; } state.waits.fetch_add(1, std::memory_order_relaxed); + const auto start = std::chrono::steady_clock::now(); while (state.sequence.load(std::memory_order_acquire) == sequence) { std::this_thread::yield(); } + state.wait_ns.fetch_add( + static_cast(std::chrono::duration_cast( + std::chrono::steady_clock::now() - start) + .count()), + std::memory_order_relaxed); } static inline DownloadLandingState state; }; -/// EDEN_NX2_READ_CHECK: the latest writes from the GPU's side that ReadDownloadsMayCover +/// TEST (EDEN_NX2_READ_CHECK): the latest writes from the GPU's side that ReadDownloadsMayCover /// let through, to name the writer when the check finds newer bytes where an earlier read's /// download landed. class ReadCheckWrites { public: [[nodiscard]] static bool Enabled() { - return AsyncReadback::ReadCheckEvery() != 0; + static const bool enabled = std::getenv("EDEN_NX2_READ_CHECK") != nullptr; + return enabled; } static void Note(DAddr addr, u64 size, const char* writer) { diff --git a/src/video_core/rasterizer_interface.h b/src/video_core/rasterizer_interface.h index bfc7954a24..c7cfa18bad 100644 --- a/src/video_core/rasterizer_interface.h +++ b/src/video_core/rasterizer_interface.h @@ -38,6 +38,11 @@ namespace VideoCommon::OotNetwork { struct BypassState; } +namespace VideoCommon::ReadTrace { +struct Read; +enum class Step : u8; +} // namespace VideoCommon::ReadTrace + namespace VideoCore { enum class LoadCallbackStage { @@ -125,8 +130,12 @@ class RasterizerInterface { /// before it reads, and the GPU's side before it writes guest memory there. virtual void FinishRead([[maybe_unused]] DAddr addr, [[maybe_unused]] u64 size) {} - /// A query result the query cache writes to guest memory later, at a fence or sync point. - /// When this returns true, FinishReport once it wrote or dropped it. + /// TEST (EDEN_NX2_READ_TRACE, nx2_read_trace.h): a step of a guest read that waits, on the + /// reading thread (GPU::OnCPURead). + virtual void TraceRead([[maybe_unused]] VideoCommon::ReadTrace::Read& read, + [[maybe_unused]] VideoCommon::ReadTrace::Step step) {} + + /// A query result written later; finish its tracking when written or discarded. virtual bool NoteReportPending([[maybe_unused]] DAddr addr, [[maybe_unused]] u64 size) { return false; } @@ -161,6 +170,40 @@ class RasterizerInterface { /// Switch 2: the front flushed this command list page with the split drained. virtual void NoteGpuWrittenCommandListSynced([[maybe_unused]] DAddr page) {} + /// Switch 2, GPU-written command list words the host GPU takes over (Nx2ListIndirect): the + /// back's slots for them. Whether this rasterizer has them. + [[nodiscard]] virtual bool Nx2ListSlotsSupported() const { + return false; + } + /// Any thread: a slot nobody holds, held for the caller; none while every slot is held. + [[nodiscard]] virtual std::optional Nx2AcquireListSlot() { + return std::nullopt; + } + /// Any thread: drops a hold of `slot` (Nx2AcquireListSlot's or one the back took). + virtual void Nx2ReleaseListSlot([[maybe_unused]] u32 slot) {} + /// The back, at a list's place: the host GPU copies the list's `count` words at + /// `device_addr` into `slot`. + virtual void Nx2SnapshotListWords([[maybe_unused]] DAddr device_addr, + [[maybe_unused]] u32 count, [[maybe_unused]] u32 slot) {} + /// The back: the word at `device_addr` was just written `value`, the placeholder of a slot + /// word. + /// While it holds it, compute dispatches that bind the word as constant buffer data get the + /// slot word there instead. + virtual void Nx2NoteOwedListWord([[maybe_unused]] DAddr device_addr, + [[maybe_unused]] u32 value) {} + /// The back: an inline QMD with grid words `x`, `y`, `z` was just written at `device_addr`. + /// Holds the slots placeholders among them name, until the next inline QMD written there. + virtual void Nx2NoteInlineQmdGrid([[maybe_unused]] DAddr device_addr, [[maybe_unused]] u32 x, + [[maybe_unused]] u32 y, [[maybe_unused]] u32 z) {} + /// The back: the words of `slots`, three each, read back from the host GPU (waits for it). + virtual void Nx2ReadListSlots([[maybe_unused]] std::span slots, + [[maybe_unused]] std::span words) {} + /// TEST (EDEN_NX2_LIST_INDIRECT_CHECK), the back, right after Nx2SnapshotListWords: the + /// words the front read as before for that list, for what the host GPU uses of the slot to + /// be compared with. + virtual void Nx2CheckListSlot([[maybe_unused]] u32 slot, + [[maybe_unused]] std::span words) {} + virtual void FlushGpuWrittenCommandListPages(std::span pages) { for (const DAddr page : pages) { FlushRegion(page, 0x1000, VideoCommon::CacheType::BufferCache); diff --git a/src/video_core/renderer_opengl/gl_buffer_cache.cpp b/src/video_core/renderer_opengl/gl_buffer_cache.cpp index fb65a132d0..d5bef8fca4 100644 --- a/src/video_core/renderer_opengl/gl_buffer_cache.cpp +++ b/src/video_core/renderer_opengl/gl_buffer_cache.cpp @@ -188,7 +188,7 @@ void BufferCacheRuntime::CopyBuffer(GLuint dst_buffer, Buffer& src_buffer, void BufferCacheRuntime::CopyBuffer(Buffer& dst_buffer, GLuint src_buffer, std::span copies, bool barrier, - bool) { + VideoCommon::UploadOrder) { CopyBuffer(dst_buffer.Handle(), src_buffer, copies, barrier); } diff --git a/src/video_core/renderer_opengl/gl_buffer_cache.h b/src/video_core/renderer_opengl/gl_buffer_cache.h index 831ec09f46..b23e2e70ab 100644 --- a/src/video_core/renderer_opengl/gl_buffer_cache.h +++ b/src/video_core/renderer_opengl/gl_buffer_cache.h @@ -13,6 +13,7 @@ #include "common/common_types.h" #include "video_core/buffer_cache/buffer_cache_base.h" #include "video_core/buffer_cache/memory_tracker_base.h" +#include "video_core/buffer_cache/upload_order.h" #include "video_core/renderer_opengl/gl_device.h" #include "video_core/renderer_opengl/gl_resource_manager.h" #include "video_core/renderer_opengl/gl_staging_buffer_pool.h" @@ -73,8 +74,9 @@ class BufferCacheRuntime { void FreeDeferredStagingBuffer(StagingBufferMap& buffer); - bool CanReorderUpload(const Buffer&, std::span) { - return false; + VideoCommon::UploadOrder DecideUploadOrder(const Buffer&, + std::span) { + return {}; } void CopyBuffer(GLuint dst_buffer, GLuint src_buffer, @@ -85,7 +87,7 @@ class BufferCacheRuntime { void CopyBuffer(Buffer& dst_buffer, GLuint src_buffer, std::span copies, bool barrier, - bool can_reorder_upload = false); + VideoCommon::UploadOrder order = {}); void CopyBuffer(Buffer& dst_buffer, Buffer& src_buffer, std::span copies, bool); diff --git a/src/video_core/renderer_opengl/gl_fence_manager.cpp b/src/video_core/renderer_opengl/gl_fence_manager.cpp index 5326172af8..3cce24c9e6 100644 --- a/src/video_core/renderer_opengl/gl_fence_manager.cpp +++ b/src/video_core/renderer_opengl/gl_fence_manager.cpp @@ -49,6 +49,8 @@ Fence FenceManagerOpenGL::CreateFence(bool is_stubbed) { void FenceManagerOpenGL::QueueFence(Fence& fence) { fence->Queue(); + // Vulkan's fences submit when queued; here the commands before the fence are flushed. + rasterizer.FlushCommands(); } bool FenceManagerOpenGL::IsFenceSignaled(Fence& fence) const { diff --git a/src/video_core/renderer_opengl/gl_rasterizer.cpp b/src/video_core/renderer_opengl/gl_rasterizer.cpp index bb9eea8c70..15a54cec6d 100644 --- a/src/video_core/renderer_opengl/gl_rasterizer.cpp +++ b/src/video_core/renderer_opengl/gl_rasterizer.cpp @@ -23,6 +23,7 @@ #include "video_core/engines/kepler_compute.h" #include "video_core/engines/maxwell_3d.h" #include "video_core/memory_manager.h" +#include "video_core/nx2_drain_check.h" #include "video_core/renderer_opengl/gl_device.h" #include "video_core/renderer_opengl/gl_query_cache.h" #include "video_core/renderer_opengl/gl_rasterizer.h" @@ -396,6 +397,11 @@ void RasterizerOpenGL::DrawTexture() { } void RasterizerOpenGL::DispatchCompute() { + // As after a draw: run the sync operations threads wait for, such as a guest read's flush, + // rather than only at the next draw. + SCOPE_EXIT { + gpu.TickWork(); + }; gpu_memory->FlushCaching(); ComputePipeline* const pipeline{shader_cache.CurrentComputePipeline()}; if (!pipeline) { @@ -441,6 +447,12 @@ void RasterizerOpenGL::ResetCounter(VideoCommon::QueryType type) { void RasterizerOpenGL::Query(GPUVAddr gpu_addr, VideoCommon::QueryType type, VideoCommon::QueryPropertiesFlags flags, u32 payload, u32 subreport) { + if (VideoCommon::Nx2DrainCheck::g_enabled && type == VideoCommon::QueryType::Payload) + [[unlikely]] { + if (const auto released = gpu_memory->GpuToCpuAddress(gpu_addr)) { + VideoCommon::Nx2DrainCheck::NoteRelease(*released); + } + } const auto query_cache_type = MaxwellToVideoCoreQuery(type); if (!query_cache_type.has_value()) { return QueryFallback(gpu_addr, type, flags, payload, subreport); diff --git a/src/video_core/renderer_vulkan/nx2_list_slots.cpp b/src/video_core/renderer_vulkan/nx2_list_slots.cpp new file mode 100644 index 0000000000..265cc42446 --- /dev/null +++ b/src/video_core/renderer_vulkan/nx2_list_slots.cpp @@ -0,0 +1,321 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#include +#include + +#include "common/logging.h" +#include "video_core/renderer_vulkan/nx2_list_slots.h" +#include "video_core/renderer_vulkan/vk_scheduler.h" +#include "video_core/vulkan_common/vulkan_memory_allocator.h" + +namespace Vulkan { + +namespace Stats = Tegra::Nx2ListIndirect::Stats; // TEST + +namespace { +/// TEST: the bytes of `words`. +template +[[nodiscard]] std::span Bytes(std::span words) { + return {reinterpret_cast(words.data()), words.size_bytes()}; +} +} // Anonymous namespace + +void Nx2ListSlots::Release(u32 slot) { + if (slot >= SlotCount) { + return; + } + if (holds[slot].fetch_sub(1, std::memory_order_acq_rel) == 0) [[unlikely]] { + holds[slot].fetch_add(1, std::memory_order_relaxed); + LOG_ERROR(Render_Vulkan, "Nx2 list slot {} released without a hold", slot); + } +} + +void Nx2ListSlots::Snapshot(VkBuffer src, u64 src_offset, u32 count, u32 slot) { + if (src == VK_NULL_HANDLE || slot >= SlotCount || count == 0 || + count > Tegra::Nx2ListIndirect::SlotWords) { + return; + } + Prepare(); + Stats::Count(Stats::Get().snapshots); // TEST + const VkDeviceSize base = VkDeviceSize{slot} * SlotBytes; + copies.clear(); + copies.push_back(VkBufferCopy{src_offset, base, VkDeviceSize{count} * 4}); + if (count == Tegra::Nx2ListIndirect::SlotWords) { + // The grid as DispatchNativeCompute takes it: height and depth are 16 bits. The upper + // halves of their words in the slot stay 0 from the fill (Prepare). + copies.push_back(VkBufferCopy{src_offset, base + DispatchOffset, 4}); + copies.push_back(VkBufferCopy{src_offset + 4, base + DispatchOffset + 4, 2}); + copies.push_back(VkBufferCopy{src_offset + 8, base + DispatchOffset + 8, 2}); + } + RecordCopies(src, *buffer, copies); +} + +void Nx2ListSlots::NoteOwed(DAddr device_addr, u32 value, u32 guest_word) { + // The upload wrote over older owed words at these bytes. + for (auto it = owed.lower_bound(device_addr >= 3 ? device_addr - 3 : 0); + it != owed.end() && it->first < device_addr + 4;) { + Release(Tegra::Nx2ListIndirect::PlaceholderSlot(it->second)); + it = owed.erase(it); + } + if (Stats::g_check && Tegra::Nx2ListIndirect::IsPlaceholder(value)) [[unlikely]] { + // TEST: the upload wrote the placeholder at once (a word it did not write is not owed). + Stats::NoteCompared(Stats::Check::Owed, Tegra::Nx2ListIndirect::PlaceholderSlot(value), + Bytes(std::span{&guest_word, 1}), Bytes(std::span{&value, 1})); + } + if (guest_word != value || !Tegra::Nx2ListIndirect::IsPlaceholder(value)) { + return; // It wrote nothing there + } + Stats::Count(Stats::Get().owed); // TEST + holds[Tegra::Nx2ListIndirect::PlaceholderSlot(value)].fetch_add(1, std::memory_order_acq_rel); + owed.emplace(device_addr, value); +} + +void Nx2ListSlots::NoteInlineQmdGrid(DAddr device_addr, u32 x, u32 y, u32 z) { + std::array slots{NoEntry, NoEntry, NoEntry}; + size_t count = 0; + for (const u32 word : {x, y, z}) { + if (!Tegra::Nx2ListIndirect::IsPlaceholder(word)) { + continue; + } + const u32 slot = Tegra::Nx2ListIndirect::PlaceholderSlot(word); + if (std::ranges::find(slots, slot) == slots.end()) { + slots[count++] = slot; + } + } + if (count == 0 && qmd_slots.empty()) { + return; + } + // The new holds first: an old hold of the same slot is dropped after them. + for (size_t i = 0; i < count; ++i) { + holds[slots[i]].fetch_add(1, std::memory_order_acq_rel); + } + const auto it = qmd_slots.find(device_addr); + if (it == qmd_slots.end()) { + if (count != 0) { + qmd_slots.emplace(device_addr, slots); + } + return; + } + for (const u32 slot : it->second) { + if (slot != NoEntry) { + Release(slot); + } + } + if (count == 0) { + qmd_slots.erase(it); + } else { + it->second = slots; + } +} + +void Nx2ListSlots::Read(std::span slots, std::span words) { + std::ranges::fill(words, 0U); + if (!buffer) { + return; + } + if (!readback) { + readback = memory_allocator.CreateBuffer( + VkBufferCreateInfo{ + .sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, + .pNext = nullptr, + .flags = 0, + .size = VkDeviceSize{ReadbackSlots} * SlotWordBytes, + .usage = VK_BUFFER_USAGE_TRANSFER_DST_BIT, + .sharingMode = VK_SHARING_MODE_EXCLUSIVE, + .queueFamilyIndexCount = 0, + .pQueueFamilyIndices = nullptr, + }, + MemoryUsage::Download); + } + for (size_t first = 0; first < slots.size(); first += ReadbackSlots) { + const size_t count = (std::min)(ReadbackSlots, slots.size() - first); + copies.clear(); + for (size_t i = 0; i < count; ++i) { + copies.push_back(VkBufferCopy{VkDeviceSize{slots[first + i]} * SlotBytes, + VkDeviceSize{i} * SlotWordBytes, SlotWordBytes}); + } + RecordCopies(*buffer, *readback, copies); + RecordHostBarrier(); + scheduler.Finish(); + readback.Invalidate(); + const size_t bytes = + (std::min)(count * SlotWordBytes, words.size_bytes() - first * SlotWordBytes); + std::memcpy(words.data() + first * Tegra::Nx2ListIndirect::SlotWords, + readback.Mapped().data(), bytes); + } + if (Stats::g_check) [[unlikely]] { + // TEST: the words read back against the list's, and what the checks copied. + for (size_t i = 0; i < slots.size() && (i + 1) * SlotWordBytes <= words.size_bytes(); ++i) { + const u32 slot = slots[i]; + if (slot < SlotCount && check_counts[slot] != 0) { + const size_t known = check_counts[slot]; + Stats::NoteCompared( + Stats::Check::Read, slot, + Bytes(words.subspan(i * Tegra::Nx2ListIndirect::SlotWords, known)), + Bytes(std::span{check_words[slot]}.first(known))); + } + } + CheckFlush(); + } +} + +void Nx2ListSlots::Prepare() { + if (buffer) { + return; + } + buffer = memory_allocator.CreateBuffer( + VkBufferCreateInfo{ + .sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, + .pNext = nullptr, + .flags = 0, + .size = VkDeviceSize{SlotCount} * SlotBytes, + .usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT | + VK_BUFFER_USAGE_INDIRECT_BUFFER_BIT, + .sharingMode = VK_SHARING_MODE_EXCLUSIVE, + .queueFamilyIndexCount = 0, + .pQueueFamilyIndices = nullptr, + }, + MemoryUsage::DeviceLocal); + scheduler.RequestOutsideRenderPassOperationContext(); + scheduler.Record([dst = *buffer](vk::CommandBuffer cmdbuf) { + static constexpr VkMemoryBarrier WRITE_BARRIER{ + .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, + .dstAccessMask = VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT, + }; + cmdbuf.FillBuffer(dst, 0, VK_WHOLE_SIZE, 0); + cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, + vk::PIPELINE_STAGE_GRAPHICS_COMPUTE_TRANSFER, 0, WRITE_BARRIER); + }); +} + +void Nx2ListSlots::RecordCopies(VkBuffer src, VkBuffer dst, std::span regions) { + static constexpr VkMemoryBarrier READ_BARRIER{ + .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_MEMORY_WRITE_BIT, + .dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT | VK_ACCESS_TRANSFER_WRITE_BIT, + }; + static constexpr VkMemoryBarrier WRITE_BARRIER{ + .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, + .dstAccessMask = VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT, + }; + scheduler.RequestOutsideRenderPassOperationContext(); + scheduler.Record([src, dst, list = std::vector(regions.begin(), regions.end())]( + vk::CommandBuffer cmdbuf) { + cmdbuf.PipelineBarrier(vk::PIPELINE_STAGE_GRAPHICS_COMPUTE_TRANSFER, + VK_PIPELINE_STAGE_TRANSFER_BIT, 0, READ_BARRIER); + cmdbuf.CopyBuffer(src, dst, list); + cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, + vk::PIPELINE_STAGE_GRAPHICS_COMPUTE_TRANSFER, 0, WRITE_BARRIER); + }); +} + +void Nx2ListSlots::RecordHostBarrier() { + scheduler.Record([](vk::CommandBuffer cmdbuf) { + // A fence makes device writes available, not visible to the host. + static constexpr VkMemoryBarrier HOST_BARRIER{ + .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, + .dstAccessMask = VK_ACCESS_HOST_READ_BIT, + }; + cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_HOST_BIT, 0, + HOST_BARRIER); + }); +} + +void Nx2ListSlots::CheckSnapshot(u32 slot, std::span words) { + if (slot >= SlotCount || words.empty() || words.size() > Tegra::Nx2ListIndirect::SlotWords || + !buffer) { + return; + } + check_words[slot] = {}; + std::ranges::copy(words, check_words[slot].begin()); + check_counts[slot] = static_cast(words.size()); + CheckRecord(Stats::Check::Snapshot, slot, *buffer, VkDeviceSize{slot} * SlotBytes, + Bytes(words)); + if (check_entries.size() >= CheckEntries / 2) { + CheckFlush(); // At the list's place on the back: between methods + } +} + +void Nx2ListSlots::CheckDispatch(u32 slot) { + if (slot >= SlotCount || check_counts[slot] != Tegra::Nx2ListIndirect::SlotWords) { + return; + } + const auto& words = check_words[slot]; + // The grid DispatchNativeCompute dispatches from the words (the width as the slot has it). + const std::array grid{words[0], words[1] & 0xffff, words[2] & 0xffff}; + if (!(grid[0] && grid[1] && grid[2])) { + Stats::Count(Stats::Get().empty_grids); + } + if (grid[0] >> 31) { + Stats::Count(Stats::Get().wide_grids); + } + CheckRecord(Stats::Check::Dispatch, slot, *buffer, + VkDeviceSize{slot} * SlotBytes + DispatchOffset, Bytes(std::span{grid})); +} + +void Nx2ListSlots::CheckPatch(VkBuffer dst) { + for (size_t i = 0; i < copies.size() && i < patch_words.size(); ++i) { + const u32 slot = Tegra::Nx2ListIndirect::PlaceholderSlot(patch_words[i].value); + const u32 word = Tegra::Nx2ListIndirect::PlaceholderWord(patch_words[i].value); + if (slot >= SlotCount || word >= check_counts[slot]) { + continue; + } + const auto bytes = Bytes(std::span{&check_words[slot][word], 1}) + .subspan(patch_words[i].skip, copies[i].size); + CheckRecord(Stats::Check::Patch, slot, dst, copies[i].dstOffset, bytes); + } +} + +void Nx2ListSlots::CheckRecord(Stats::Check check, u32 slot, VkBuffer src, VkDeviceSize src_offset, + std::span expected) { + if (check_entries.size() >= CheckEntries || expected.size() > CheckEntryBytes) { + Stats::Count(Stats::Get().unchecked); + return; + } + if (!check_buffer) { + check_buffer = memory_allocator.CreateBuffer( + VkBufferCreateInfo{ + .sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, + .pNext = nullptr, + .flags = 0, + .size = VkDeviceSize{CheckEntries} * CheckEntryBytes, + .usage = VK_BUFFER_USAGE_TRANSFER_DST_BIT, + .sharingMode = VK_SHARING_MODE_EXCLUSIVE, + .queueFamilyIndexCount = 0, + .pQueueFamilyIndices = nullptr, + }, + MemoryUsage::Download); + } + CheckEntry entry{check, slot, static_cast(expected.size()), {}}; + std::memcpy(entry.expected.data(), expected.data(), expected.size()); + const VkBufferCopy copy{src_offset, VkDeviceSize{check_entries.size()} * CheckEntryBytes, + expected.size()}; + check_entries.push_back(entry); + RecordCopies(src, *check_buffer, std::span{©, 1}); +} + +void Nx2ListSlots::CheckFlush() { + if (check_entries.empty()) { + return; + } + RecordHostBarrier(); + scheduler.Finish(); + check_buffer.Invalidate(); + const std::span data = std::as_const(check_buffer).Mapped(); + for (size_t i = 0; i < check_entries.size(); ++i) { + const CheckEntry& entry = check_entries[i]; + Stats::NoteCompared(entry.check, entry.slot, data.subspan(i * CheckEntryBytes, entry.bytes), + std::span{entry.expected}.first(entry.bytes)); + } + check_entries.clear(); +} + +} // namespace Vulkan diff --git a/src/video_core/renderer_vulkan/nx2_list_slots.h b/src/video_core/renderer_vulkan/nx2_list_slots.h new file mode 100644 index 0000000000..c9cc6d4c6d --- /dev/null +++ b/src/video_core/renderer_vulkan/nx2_list_slots.h @@ -0,0 +1,205 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common/common_types.h" +#include "video_core/nx2_list_indirect.h" +#include "video_core/nx2_list_indirect_stats.h" +#include "video_core/vulkan_common/vulkan_wrapper.h" + +namespace Vulkan { + +class MemoryAllocator; +class Scheduler; + +/// The split GPU thread's back's slots for GPU-written command list words the host GPU takes +/// over (Nx2ListIndirect). A slot is SlotBytes of device-local memory: the list's words as the +/// back found them at the list's place (the snapshot, raw), and at DispatchOffset the same three +/// as DispatchNativeCompute takes a grid: width, height & 0xffff, depth & 0xffff (a +/// VkDispatchIndirectCommand). Holds keep a slot from being used again: the front's while its +/// macros' words hold the slot's placeholders, the back's while a QMD's grid or a word compute +/// dispatches bind as constant buffer data holds them. +/// +/// The words the front's inline uploads put placeholders in are owed words: while one holds its +/// placeholder in guest memory, a compute dispatch that binds it as constant buffer data gets the +/// slot word there (PatchUniform, recorded after the binding filled its buffer). +class Nx2ListSlots { +public: + static constexpr u32 SlotCount = Tegra::Nx2ListIndirect::SlotCount; + static constexpr u32 SlotBytes = 32; + static constexpr u32 DispatchOffset = 16; + + explicit Nx2ListSlots(MemoryAllocator& memory_allocator_, Scheduler& scheduler_) + : memory_allocator{memory_allocator_}, scheduler{scheduler_} {} + + /// Any thread (the front's): a slot nobody holds, held for the caller. + [[nodiscard]] std::optional Acquire() { + for (u32 i = 0; i < SlotCount; ++i) { + const u32 slot = (next + i) % SlotCount; + u32 expected = 0; + if (holds[slot].compare_exchange_strong(expected, 1, std::memory_order_acq_rel)) { + next = slot + 1; + return slot; + } + } + return std::nullopt; + } + + /// Any thread: drops a hold of `slot`. + void Release(u32 slot); + + [[nodiscard]] bool Held(u32 slot) const { + return slot < SlotCount && holds[slot].load(std::memory_order_acquire) != 0; + } + + /// The back: records the copy of the `count` words at `src_offset` of `src` (the list's host + /// buffer, the copy's use of the words marked) into `slot`. + void Snapshot(VkBuffer src, u64 src_offset, u32 count, u32 slot); + + /// The back: the word at `device_addr` was just written `value` (a placeholder) by an inline + /// upload; `guest_word` is what guest memory holds there now. + void NoteOwed(DAddr device_addr, u32 value, u32 guest_word); + + /// The back: an inline QMD with grid words x, y, z was just written at `device_addr`. + void NoteInlineQmdGrid(DAddr device_addr, u32 x, u32 y, u32 z); + + [[nodiscard]] bool HasOwed() const { + return !owed.empty(); + } + + /// The back, binding a compute dispatch's constant buffer [device_addr, +size) at `offset` + /// of `dst`, filled already: records copies of the slot words owed there over it. `read` + /// gives the word guest memory holds at a device address: a word that no longer holds its + /// placeholder is no longer owed. + template + void PatchUniform(DAddr device_addr, u32 size, VkBuffer dst, u64 offset, Read&& read) { + if (dst == VK_NULL_HANDLE || size == 0) { + return; + } + copies.clear(); + patch_words.clear(); // TEST + const DAddr end = device_addr + size; + for (auto it = owed.lower_bound(device_addr >= 3 ? device_addr - 3 : 0); + it != owed.end() && it->first < end;) { + const DAddr word = it->first; + const u32 value = it->second; + if (read(word) != value) { + Tegra::Nx2ListIndirect::Stats::Count( + Tegra::Nx2ListIndirect::Stats::Get().owed_dropped); // TEST + Release(Tegra::Nx2ListIndirect::PlaceholderSlot(value)); + it = owed.erase(it); + continue; + } + const DAddr first = (std::max)(word, device_addr); + const DAddr last = (std::min)(word + 4, end); + if (first < last) { + const VkDeviceSize source = + VkDeviceSize{Tegra::Nx2ListIndirect::PlaceholderSlot(value)} * SlotBytes + + VkDeviceSize{Tegra::Nx2ListIndirect::PlaceholderWord(value)} * 4 + + (first - word); + copies.push_back( + VkBufferCopy{source, offset + (first - device_addr), last - first}); + patch_words.push_back({value, static_cast(first - word)}); // TEST + } + ++it; + } + if (!copies.empty()) { + ++patched_binds; + Tegra::Nx2ListIndirect::Stats::Count( + Tegra::Nx2ListIndirect::Stats::Get().patched_binds); // TEST + RecordCopies(*buffer, dst, copies); + if (Tegra::Nx2ListIndirect::Stats::g_check) [[unlikely]] { + CheckPatch(dst); + } + } + } + + /// The buffer and offset of `slot`'s VkDispatchIndirectCommand. + [[nodiscard]] std::pair DispatchArguments(u32 slot) const { + return {*buffer, VkDeviceSize{slot} * SlotBytes + DispatchOffset}; + } + + /// The back: the raw words of `slots`, three each, once the host GPU wrote them (submits and + /// waits for it). + void Read(std::span slots, std::span words); + + /// Dispatches whose constant buffers got owed words. + [[nodiscard]] u64 PatchedBinds() const { + return patched_binds; + } + + /// TEST (EDEN_NX2_LIST_INDIRECT_CHECK): the words the front read as before for the list + /// whose snapshot `slot` just got. What the host GPU then uses of the slot is compared with + /// them once it ran (CheckFlush): the snapshot, the dispatches from it, the patches. + void CheckSnapshot(u32 slot, std::span words); + /// TEST: the dispatch from `slot` just recorded. + void CheckDispatch(u32 slot); + +private: + static constexpr u32 NoEntry = ~0U; + static constexpr u32 SlotWordBytes = Tegra::Nx2ListIndirect::SlotWords * sizeof(u32); + static constexpr u32 ReadbackSlots = 256; + + /// The slots' buffer, created on first use, zero-filled. + void Prepare(); + + /// Records `regions` from `src` to `dst` outside a render pass, between barriers as + /// BufferCacheRuntime::CopyBuffer puts them: after every write and read before (a slot a copy + /// writes may be read by an earlier dispatch or copy), before every read and write after. + void RecordCopies(VkBuffer src, VkBuffer dst, std::span regions); + + /// Records a barrier that makes transfer writes visible to the host, after a fence wait. + void RecordHostBarrier(); + + // TEST (EDEN_NX2_LIST_INDIRECT_CHECK): copies of what the host GPU used, compared with what + // the old path would have used once it ran them. + struct CheckEntry { + Tegra::Nx2ListIndirect::Stats::Check check; + u32 slot; + u32 bytes; + std::array expected; + }; + struct PatchWord { + u32 value; ///< The owed placeholder + u32 skip; ///< Bytes of it before the binding + }; + static constexpr u32 CheckEntries = 1024; + static constexpr u32 CheckEntryBytes = 16; + /// Records a copy of the `expected.size()` bytes at `src_offset` of `src`, for CheckFlush. + void CheckRecord(Tegra::Nx2ListIndirect::Stats::Check check, u32 slot, VkBuffer src, + VkDeviceSize src_offset, std::span expected); + /// The patch just recorded into `dst` (copies, patch_words). + void CheckPatch(VkBuffer dst); + /// Waits for the host GPU and compares what CheckRecord copied. + void CheckFlush(); + + MemoryAllocator& memory_allocator; + Scheduler& scheduler; + vk::Buffer buffer; + vk::Buffer readback; ///< Read's, host-visible + std::array, SlotCount> holds{}; + u32 next{}; ///< Where Acquire looks first + std::map owed; ///< By device address (the back's) + std::unordered_map> qmd_slots; ///< By QMD device address + std::vector copies; + u64 patched_binds{}; + // TEST (EDEN_NX2_LIST_INDIRECT_CHECK). + std::vector patch_words; + vk::Buffer check_buffer; + std::vector check_entries; + std::array, SlotCount> check_words{}; + std::array check_counts{}; ///< Words of check_words known (0: none) +}; + +} // namespace Vulkan diff --git a/src/video_core/renderer_vulkan/pipeline_optimization.h b/src/video_core/renderer_vulkan/pipeline_optimization.h index 33a2fdd790..8d88ef329a 100644 --- a/src/video_core/renderer_vulkan/pipeline_optimization.h +++ b/src/video_core/renderer_vulkan/pipeline_optimization.h @@ -12,9 +12,10 @@ namespace Vulkan { inline bool BuildsUnoptimized(VkDriverId driver_id, u64 shader_hash) { // The Legend of Zelda: Ocarina of Time (0400486024DEC000), driver 2.0.353 on an RX 6600 XT - static constexpr std::array amd_proprietary{ + static constexpr std::array amd_proprietary{ 0xeff3ccbe1438679dULL, // compute shader built as the AV1 intro starts: 6 min (#200) 0xf9ea14a22394f5b1ULL, // compute shader built as the AV1 intro starts: 18 min (#200) + 0xe0a992208d87eeb8ULL, // Kokiri Forest compute shader: 1.3-4 s stalls (RX 9070 XT, 2.0.416) }; return driver_id == VK_DRIVER_ID_AMD_PROPRIETARY && std::ranges::find(amd_proprietary, shader_hash) != amd_proprietary.end(); diff --git a/src/video_core/renderer_vulkan/present/lsfg_mipmaps.cpp b/src/video_core/renderer_vulkan/present/lsfg_mipmaps.cpp index 6454ca0480..3455557d1d 100644 --- a/src/video_core/renderer_vulkan/present/lsfg_mipmaps.cpp +++ b/src/video_core/renderer_vulkan/present/lsfg_mipmaps.cpp @@ -36,7 +36,7 @@ LsfgMipmaps::LsfgMipmaps(const Device& device, MemoryAllocator& memory_allocator {{1, VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER}, {1, VK_DESCRIPTOR_TYPE_SAMPLER}, {1, VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE}, - {LSFG_MIP_LEVELS, VK_DESCRIPTOR_TYPE_STORAGE_IMAGE}}); + {static_cast(LSFG_MIP_LEVELS), VK_DESCRIPTOR_TYPE_STORAGE_IMAGE}}); const VkExtent2D input_extent = (*frames)[0].Extent(); flow_extent = VkExtent2D{ diff --git a/src/video_core/renderer_vulkan/submission_policy.h b/src/video_core/renderer_vulkan/submission_policy.h index a109edc713..dc8566cb85 100644 --- a/src/video_core/renderer_vulkan/submission_policy.h +++ b/src/video_core/renderer_vulkan/submission_policy.h @@ -11,6 +11,8 @@ namespace Vulkan::SubmissionPolicy { inline constexpr u32 DefaultNativeBudget = 128; +/// Draws per submission on AMD's proprietary driver (see RasterizerVulkan::FlushWork). +inline constexpr u32 AmdNativeBudget = 512; enum class Action { None, diff --git a/src/video_core/renderer_vulkan/upload_batch.h b/src/video_core/renderer_vulkan/upload_batch.h new file mode 100644 index 0000000000..3bb88b8a67 --- /dev/null +++ b/src/video_core/renderer_vulkan/upload_batch.h @@ -0,0 +1,194 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include +#include + +#include "common/common_types.h" +#include "video_core/vulkan_common/vulkan_wrapper.h" + +namespace Vulkan { + +/// Uploads in order (Scheduler::RecordUpload) waiting to be recorded as one command: their +/// copies run between one barrier before and one after. Copies of one batch may run in any +/// order and at once, so a batch never holds two copies that write the same bytes, or one that +/// writes a buffer another reads. +class UploadBatch { +public: + /// Copies with the same source and destination, consecutive in `copies`. + struct Group { + VkBuffer src; + VkBuffer dst; + u32 first; + u32 count; + }; + + /// More copies than this go in a later batch. It bounds CanAdd's checks on the recording + /// thread, at the cost of two barriers per this many uploads. + static constexpr size_t MaxCopies = 64; + + [[nodiscard]] bool Empty() const noexcept { + return copies.empty(); + } + + /// Whether these copies can join the batch: it has room, they read no buffer a copy in it + /// writes and write none it reads, and they write no byte of dst that a copy in it writes. + /// Otherwise the batch is recorded first. + [[nodiscard]] bool CanAdd(VkBuffer src, VkBuffer dst, + std::span new_copies) const { + if (copies.size() + new_copies.size() > MaxCopies) { + return false; + } + for (const Group& group : groups) { + if (src == group.dst || dst == group.src) { + return false; + } + if (group.dst != dst) { + continue; + } + for (u32 index = group.first; index < group.first + group.count; ++index) { + const VkBufferCopy& pending = copies[index]; + for (const VkBufferCopy& copy : new_copies) { + if (copy.dstOffset < pending.dstOffset + pending.size && + pending.dstOffset < copy.dstOffset + copy.size) { + return false; + } + } + } + } + return true; + } + + /// Records a batch: the barrier before an in-order buffer copy (BufferCacheRuntime:: + /// CopyBuffer's), the copies, and the barrier after one, made to reach transfers as well. + /// Recorded one by one, the barriers between two uploads ordered the copies before them + /// ahead of the copies after them, which CanAdd makes needless, and ahead of all later work: + /// graphics and compute through the barrier after a copy, transfers (in later submissions + /// too) through the barrier before the next one. The barrier after the batch does both. + static void Record(vk::CommandBuffer cmdbuf, std::span batch_copies, + std::span batch_groups) { + static constexpr VkMemoryBarrier READ_BARRIER{ + .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_MEMORY_WRITE_BIT, + .dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT | VK_ACCESS_TRANSFER_WRITE_BIT, + }; + static constexpr VkMemoryBarrier WRITE_BARRIER{ + .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, + .dstAccessMask = VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT, + }; + cmdbuf.PipelineBarrier(vk::PIPELINE_STAGE_GRAPHICS_COMPUTE_TRANSFER, + VK_PIPELINE_STAGE_TRANSFER_BIT, 0, READ_BARRIER); + for (const Group& group : batch_groups) { + cmdbuf.CopyBuffer(group.src, group.dst, + vk::Span(batch_copies.data() + group.first, + group.count)); + } + cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, + vk::PIPELINE_STAGE_GRAPHICS_COMPUTE_TRANSFER, 0, WRITE_BARRIER); + } + + void Add(VkBuffer src, VkBuffer dst, std::span new_copies) { + const u32 first = static_cast(copies.size()); + const u32 count = static_cast(new_copies.size()); + copies.insert(copies.end(), new_copies.begin(), new_copies.end()); + if (!groups.empty() && groups.back().src == src && groups.back().dst == dst) { + groups.back().count += count; + } else { + groups.push_back({src, dst, first, count}); + } + } + + std::vector copies; + std::vector groups; +}; + +/// Uploads that run in the upload command buffer, ahead of all the work of their submission's +/// main command buffer (Scheduler::RecordReorderedUpload). They are recorded as one command when +/// the submission is: a copy per source and destination. Nothing orders them against each other +/// and nothing has to: the buffer cache keeps an upload in order when work recorded before it in +/// the submission, an earlier upload included, uses a byte it writes (DecideUploadOrder). +class ReorderedUploadBatch { +public: + [[nodiscard]] bool Empty() const noexcept { + return entries.empty(); + } + + /// prior_tick: the newest earlier submission whose work uses bytes the copies write, or a + /// later one; 0 if none. + void Add(VkBuffer src, VkBuffer dst, std::span new_copies, u64 prior_tick) { + for (const VkBufferCopy& copy : new_copies) { + entries.push_back(Entry{src, dst, copy}); + } + newest_prior_tick = (std::max)(newest_prior_tick, prior_tick); + } + + /// Empties the batch into its copies, grouped by source and destination (in the order they + /// were added within a group), and the newest earlier submission they must come after. + void Take(std::vector& copies, std::vector& groups, + u64& prior_tick) { + std::stable_sort(entries.begin(), entries.end(), [](const Entry& lhs, const Entry& rhs) { + if (lhs.src != rhs.src) { + return std::less{}(lhs.src, rhs.src); + } + return std::less{}(lhs.dst, rhs.dst); + }); + copies.clear(); + groups.clear(); + for (const Entry& entry : entries) { + const u32 index = static_cast(copies.size()); + copies.push_back(entry.copy); + if (!groups.empty() && groups.back().src == entry.src && + groups.back().dst == entry.dst) { + ++groups.back().count; + } else { + groups.push_back(UploadBatch::Group{entry.src, entry.dst, index, 1}); + } + } + prior_tick = newest_prior_tick; + entries.clear(); + newest_prior_tick = 0; + } + + /// Records a batch into the upload command buffer. With after_prior_work it begins with a + /// barrier whose first scope is all the work submitted before it, earlier submissions' main + /// command buffers included: their reads and writes of the bytes the copies write come first. + /// The upload command buffer's last barrier (Scheduler::SubmitExecution) orders the copies + /// before the main command buffer's work and all later work. + static void Record(vk::CommandBuffer upload_cmdbuf, std::span batch_copies, + std::span batch_groups, bool after_prior_work) { + static constexpr VkMemoryBarrier PRIOR_WORK_BARRIER{ + .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_MEMORY_WRITE_BIT, + .dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT | VK_ACCESS_TRANSFER_WRITE_BIT, + }; + if (after_prior_work) { + upload_cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_TRANSFER_BIT, 0, PRIOR_WORK_BARRIER); + } + for (const UploadBatch::Group& group : batch_groups) { + upload_cmdbuf.CopyBuffer( + group.src, group.dst, + vk::Span(batch_copies.data() + group.first, group.count)); + } + } + +private: + struct Entry { + VkBuffer src; + VkBuffer dst; + VkBufferCopy copy; + }; + + std::vector entries; + u64 newest_prior_tick = 0; +}; + +} // namespace Vulkan diff --git a/src/video_core/renderer_vulkan/vk_buffer_cache.cpp b/src/video_core/renderer_vulkan/vk_buffer_cache.cpp index a87fbe02df..a8fab2165a 100644 --- a/src/video_core/renderer_vulkan/vk_buffer_cache.cpp +++ b/src/video_core/renderer_vulkan/vk_buffer_cache.cpp @@ -6,11 +6,16 @@ #include #include +#include #include #include +#include #include +#include #include +#include + #include "video_core/buffer_cache/buffer_cache_base.h" #include "video_core/nx2_perf.h" #include "video_core/nx2_profile.h" @@ -18,6 +23,7 @@ #include "video_core/renderer_vulkan/vertex_repack.h" #include "video_core/renderer_vulkan/maxwell_to_vk.h" +#include "video_core/renderer_vulkan/vk_gpu_cost.h" #include "video_core/renderer_vulkan/vk_scheduler.h" #include "video_core/renderer_vulkan/vk_staging_buffer_pool.h" #include "video_core/renderer_vulkan/vk_update_descriptor.h" @@ -28,6 +34,9 @@ namespace Vulkan { namespace { +/// TEST (EDEN_NX2_GPU_COST=2): Buffer::MarkUsage notes page ticks, DecideUploadOrder classifies. +const bool g_trace_uploads = GpuCost::TracesUploads(); + void HostReadBarrier(vk::CommandBuffer cmdbuf) { const VkMemoryBarrier barrier{ .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, @@ -107,7 +116,7 @@ vk::Buffer CreateBuffer(const Device& device, const MemoryAllocator& memory_allo } // Anonymous namespace Buffer::Buffer(BufferCacheRuntime& runtime, VideoCommon::NullBufferParams null_params) - : VideoCommon::BufferBase(null_params), scheduler{&runtime.scheduler}, tracker{4096} { + : VideoCommon::BufferBase(null_params), scheduler{&runtime.scheduler} { if (runtime.device.HasNullDescriptor()) { return; } @@ -124,8 +133,7 @@ Buffer::Buffer(BufferCacheRuntime& runtime, DAddr cpu_addr_, u64 size_bytes_, : VideoCommon::BufferBase(cpu_addr_, size_bytes_), device{&runtime.device}, scheduler{&runtime.scheduler}, buffer{CreateBuffer(*device, runtime.memory_allocator, SizeBytes(), - runtime.SparseAlignmentFor(sparse_compatible_))}, - tracker{SizeBytes()} { + runtime.SparseAlignmentFor(sparse_compatible_))} { sparse_compatible = sparse_compatible_; if (runtime.device.HasDebuggingToolAttached()) { buffer.SetObjectNameEXT(fmt::format("Buffer {:#x}", CpuAddr()).c_str()); @@ -136,8 +144,43 @@ Buffer::Buffer(BufferCacheRuntime& runtime, DAddr cpu_addr_, u64 size_bytes_, } void Buffer::MarkUsage(u64 offset, u64 size) noexcept { - tracker.Track(offset, size); - last_usage_tick = scheduler->CurrentTick(); + const u64 tick = scheduler->CurrentTick(); + uses.Track(offset, size, tick); + if (UploadReorderCheck* const check = UploadReorderCheck::Active()) [[unlikely]] { + check->NoteUse(*buffer, offset, size, UploadReorderCheck::Source::Marked); + } + if (g_trace_uploads) [[unlikely]] { + const std::array use{offset, size, tick}; + if (std::ranges::find(noted_uses, use) != noted_uses.end() || size == 0) { + return; + } + noted_uses[noted_cursor++ % noted_uses.size()] = use; + if (use_ticks.empty()) { + use_ticks.resize(static_cast(SizeBytes() >> 12) + 1); + } + const size_t last = (std::min)(static_cast((offset + size - 1) >> 12), + use_ticks.size() - 1); + for (size_t page = static_cast(offset >> 12); page <= last; ++page) { + use_ticks[page] = tick; + } + } +} + +void Buffer::TrackUploadWrite(u64 offset, u64 size) noexcept { + uses.Track(offset, size, scheduler->CurrentTick()); +} + +u64 Buffer::NewestUseTick(u64 offset, u64 size) const noexcept { + u64 newest = 0; + if (size == 0 || use_ticks.empty()) { + return newest; + } + const size_t last = + (std::min)(static_cast((offset + size - 1) >> 12), use_ticks.size() - 1); + for (size_t page = static_cast(offset >> 12); page <= last; ++page) { + newest = (std::max)(newest, use_ticks[page]); + } + return newest; } VkBufferView Buffer::View(u32 offset, u32 size, VideoCore::Surface::PixelFormat format) { @@ -382,7 +425,7 @@ BufferCacheRuntime::BufferCacheRuntime(const Device& device_, MemoryAllocator& m staging_pool{staging_pool_}, guest_descriptor_queue{guest_descriptor_queue_}, quad_index_pass(device, scheduler, descriptor_pool, staging_pool, compute_pass_descriptor_queue), - multi_range_buffers(device_) { + multi_range_buffers(device_), reorder_check{UploadReorderCheck::Create(scheduler_)} { const VkDriverIdKHR driver_id = device.GetDriverID(); const bool enable_vertex_repack = VertexRepackEnabled(); LOG_INFO(Render_Vulkan, "69-byte vertex layout repacking {} ({})", @@ -463,13 +506,10 @@ u64 BufferCacheRuntime::GetMaxStorageBufferRange() const { return device.GetMaxStorageBufferRange(); } -void BufferCacheRuntime::TickFrame(Common::SlotVector& slot_buffers) noexcept { +void BufferCacheRuntime::TickFrame( + [[maybe_unused]] Common::SlotVector& slot_buffers) noexcept { + // Buffers' uses need no reset: they are kept by submission (VideoCommon::SubmissionUses). multi_range_buffers.TickFrame(scheduler); - for (auto it = slot_buffers.begin(); it != slot_buffers.end(); it++) { - if (scheduler.IsFree(it->LastUsageTick())) { - it->ResetUsageTracking(); - } - } } u64 BufferCacheRuntime::CurrentTick() { @@ -489,11 +529,17 @@ void BufferCacheRuntime::WaitSubmittedTick(u64 tick) { if (master.IsFree(tick)) { return; } + // The split's front waits for the GPU: the back runs meanwhile what the front sent, also the + // entries since the front's last publish, which it does not see yet (fewer than 256 words). + auto& split = VideoCommon::GpuSplit::Get(); + const bool front = !VideoCommon::GpuSplit::OnBackThread() && split.Active(); + if (front) { + split.Publish(); + } // The copy's submission may be deferred (Scheduler::DeferFlush): the split's front asks the // back for it rather than waiting for the back's next submission. If the back has stopped, // nothing else uses the scheduler and Wait submits it. - if (!scheduler.IsSubmitted(tick) && VideoCommon::GpuSplit::Get().Active() && - !VideoCommon::GpuSplit::OnBackThread()) { + if (!scheduler.IsSubmitted(tick) && front) { VideoCommon::Nx2Profile::Scope scope{VideoCommon::Nx2Profile::Zone::ListSubmitWait}; if (!scheduler.WaitSubmitted(tick)) { scheduler.Wait(tick); @@ -555,24 +601,96 @@ bool BufferCacheRuntime::IsFreeNow(u64 tick) { return scheduler.IsFree(tick); } -bool BufferCacheRuntime::CanReorderUpload(const Buffer& buffer, - std::span copies) { - if (Settings::values.disable_buffer_reorder) { - return false; +u64 BufferCacheRuntime::TraceMark() { + return scheduler.TraceMark(); +} + +VideoCommon::ReadTrace::Counts BufferCacheRuntime::TraceCounts() const { + return scheduler.TraceCounts(); +} + +void BufferCacheRuntime::TraceCountWork(bool dispatch) { + scheduler.TraceCountWork(dispatch); +} + +VideoCommon::UploadOrder BufferCacheRuntime::DecideUploadOrder( + Buffer& buffer, std::span copies) { + VideoCommon::UploadOrder order{}; + if (!Settings::values.disable_buffer_reorder) { + order = VideoCommon::DecideUploadOrder(buffer.Uses(), copies, scheduler.CurrentTick()); } - const bool can_use_upload_cmdbuf = - std::ranges::all_of(copies, [&](const VideoCommon::BufferCopy& copy) { - return !buffer.IsRegionUsed(copy.dst_offset, copy.size); - }); - return can_use_upload_cmdbuf; + if (g_trace_uploads) [[unlikely]] { + TraceUpload(buffer, copies, order.reorder); + } + // An upload of these bytes later in this submission must run after this one: in order. + for (const VideoCommon::BufferCopy& copy : copies) { + buffer.TrackUploadWrite(copy.dst_offset, copy.size); + } + return order; +} + +void BufferCacheRuntime::TraceUpload(const Buffer& buffer, + std::span copies, + bool reordered) { + // TEST (EDEN_NX2_GPU_COST): where the destination's last use was when an upload has to run + // in order (main command buffer, ends the render pass). The usage bits only clear when the + // whole buffer is idle at a frame's end (TickFrame). + enum Class : size_t { Reordered, LastUseDone, InFlight, ThisCommandBuffer, NoTick, Count }; + static constexpr std::array names{ + "reorderable (upload command buffer)", "in order, last use of its pages done", + "in order, last use submitted, not done", "in order, used in this command buffer", + "in order, no use noted"}; + struct Totals { + std::array count{}; + std::array bytes{}; + std::chrono::steady_clock::time_point start = std::chrono::steady_clock::now(); + }; + static std::mutex mutex; + static Totals totals; + u64 newest = 0; + u64 bytes = 0; + for (const VideoCommon::BufferCopy& copy : copies) { + newest = (std::max)(newest, buffer.NewestUseTick(copy.dst_offset, copy.size)); + bytes += copy.size; + } + const auto done = [&] { + if (scheduler.IsFree(newest)) { + return true; + } + scheduler.GetMasterSemaphore().Refresh(); + return scheduler.IsFree(newest); + }; + const Class kind = reordered ? Reordered + : newest == 0 ? NoTick + : newest >= scheduler.CurrentTick() ? ThisCommandBuffer + : !done() ? InFlight + : LastUseDone; + std::scoped_lock lock{mutex}; + ++totals.count[kind]; + totals.bytes[kind] += bytes; + const auto now = std::chrono::steady_clock::now(); + if (now - totals.start < std::chrono::seconds{10}) { + return; + } + const double seconds = std::chrono::duration(now - totals.start).count(); + std::string line; + for (size_t i = 0; i < Count; ++i) { + line += fmt::format("{}{}: {:.0f}/s {:.0f} KiB/s", i == 0 ? "" : ", ", names[i], + static_cast(totals.count[i]) / seconds, + static_cast(totals.bytes[i]) / 1024.0 / seconds); + } + LOG_INFO(Render_Vulkan, "Uploads in {:.1f} s: {}", seconds, line); + totals = {}; } void BufferCacheRuntime::CopyBuffer(VkBuffer dst_buffer, VkBuffer src_buffer, std::span copies, bool barrier, - bool can_reorder_upload) { + VideoCommon::UploadOrder order, std::source_location location) { if (dst_buffer == VK_NULL_HANDLE || src_buffer == VK_NULL_HANDLE) { return; } + const bool from_stream = src_buffer == staging_pool.StreamBuf(); + const bool reorder = from_stream && order.reorder; static constexpr VkMemoryBarrier READ_BARRIER{ .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, .pNext = nullptr, @@ -586,29 +704,59 @@ void BufferCacheRuntime::CopyBuffer(VkBuffer dst_buffer, VkBuffer src_buffer, .dstAccessMask = VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT, }; + if (reorder_check) [[unlikely]] { + // The upload's place against the uses before it, then the copy's own uses. + if (from_stream) { + reorder_check->CheckUpload(dst_buffer, copies, + reorder ? order : VideoCommon::UploadOrder{}, location); + } + for (const VideoCommon::BufferCopy& copy : copies) { + if (!from_stream) { + reorder_check->NoteUse(src_buffer, copy.src_offset, copy.size, + UploadReorderCheck::Source::CopyRead); + } + reorder_check->NoteUse(dst_buffer, copy.dst_offset, copy.size, + UploadReorderCheck::Source::CopyWrite); + } + } + // Measuring a popular game, this number never exceeds the specified size once data is warmed up boost::container::small_vector vk_copies(copies.size()); std::ranges::transform(copies, vk_copies.begin(), MakeBufferCopy); - if (src_buffer == staging_pool.StreamBuf() && can_reorder_upload) { - scheduler.RecordWithUploadBuffer([src_buffer, dst_buffer, vk_copies]( - vk::CommandBuffer, vk::CommandBuffer upload_cmdbuf) { - upload_cmdbuf.CopyBuffer(src_buffer, dst_buffer, vk_copies); - }); + if (reorder) { + // Ahead of this submission's work: no render pass ends for it. Recorded into the upload + // command buffer with the submission's other reordered uploads when it is submitted. + scheduler.RecordReorderedUpload( + src_buffer, dst_buffer, + std::span(vk_copies.data(), vk_copies.size()), order.prior_tick); return; } - scheduler.RequestOutsideRenderPassOperationContext(); - scheduler.Record([src_buffer, dst_buffer, vk_copies, barrier](vk::CommandBuffer cmdbuf) { - if (barrier) { - cmdbuf.PipelineBarrier(vk::PIPELINE_STAGE_GRAPHICS_COMPUTE_TRANSFER, - VK_PIPELINE_STAGE_TRANSFER_BIT, 0, READ_BARRIER); - } - cmdbuf.CopyBuffer(src_buffer, dst_buffer, vk_copies); - if (barrier) { - cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, - vk::PIPELINE_STAGE_GRAPHICS_COMPUTE, 0, WRITE_BARRIER); - } - }); + scheduler.RequestOutsideRenderPassOperationContext(location); + if (barrier && from_stream && !vk_copies.empty()) { + // An upload in order, with these barriers. Runs of them (Ocarina of Time's inline writes + // into GPU-written pages: ~600 a frame) go in one command between one pair of them. + scheduler.RecordUpload(src_buffer, dst_buffer, + std::span(vk_copies.data(), vk_copies.size()), + location); + return; + } + scheduler.Record( + [src_buffer, dst_buffer, vk_copies, barrier](vk::CommandBuffer cmdbuf) { + if (barrier) { + cmdbuf.PipelineBarrier(vk::PIPELINE_STAGE_GRAPHICS_COMPUTE_TRANSFER, + VK_PIPELINE_STAGE_TRANSFER_BIT, 0, READ_BARRIER); + } + cmdbuf.CopyBuffer(src_buffer, dst_buffer, vk_copies); + if (barrier) { + // Later transfers too: a reordered upload of the same bytes in a later + // submission has no barrier of its own before it. + cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, + vk::PIPELINE_STAGE_GRAPHICS_COMPUTE_TRANSFER, 0, + WRITE_BARRIER); + } + }, + location); } void BufferCacheRuntime::PreCopyBarrier() { @@ -643,6 +791,7 @@ void BufferCacheRuntime::ClearBuffer(VkBuffer dest_buffer, u32 offset, size_t si if (dest_buffer == VK_NULL_HANDLE) { return; } + NoteCheckUse(dest_buffer, offset, size, UploadReorderCheck::Source::Clear); static constexpr VkMemoryBarrier READ_BARRIER{ .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, .pNext = nullptr, @@ -700,6 +849,7 @@ bool BufferCacheRuntime::BindMultiRangeStorageBuffer(u64 key, bool is_written) { void BufferCacheRuntime::BindIndexBuffer(PrimitiveTopology topology, IndexFormat index_format, u32 base_vertex, u32 num_indices, VkBuffer buffer, u32 offset, [[maybe_unused]] u32 size) { + NoteCheckUse(buffer, offset, size, UploadReorderCheck::Source::Index); VkIndexType vk_index_type = MaxwellToVK::IndexFormat(index_format); VkDeviceSize vk_offset = offset; VkBuffer vk_buffer = buffer; @@ -775,6 +925,7 @@ void BufferCacheRuntime::BindVertexBuffer(u32 index, VkBuffer buffer, u32 offset if (index >= device.GetMaxVertexInputBindings()) { return; } + NoteCheckUse(buffer, offset, size, UploadReorderCheck::Source::Vertex); if (buffer != VK_NULL_HANDLE && size != 0 && IsVertexRepacked(index)) { const auto repacked = vertex_repack_pass->Assemble(buffer, offset, size, u64{offset} + size); @@ -809,6 +960,8 @@ void BufferCacheRuntime::BindVertexBuffers(VideoCommon::HostBindings& bi boost::container::static_vector buffer_handles(bindings.buffers.size()); for (u32 i = 0; i < bindings.buffers.size(); ++i) { auto handle = bindings.buffers[i]->Handle(); + NoteCheckUse(handle, bindings.offsets[i], bindings.sizes[i], + UploadReorderCheck::Source::Vertex); if (handle == VK_NULL_HANDLE) { bindings.offsets[i] = 0; bindings.sizes[i] = VK_WHOLE_SIZE; @@ -862,6 +1015,7 @@ void BufferCacheRuntime::BindTransformFeedbackBuffer(u32 index, VkBuffer buffer, // Already logged in the rasterizer return; } + NoteCheckUse(buffer, offset, size, UploadReorderCheck::Source::TransformFeedback); if (buffer == VK_NULL_HANDLE) { // Vulkan doesn't support null transform feedback buffers. // Replace it with our own null buffer. @@ -885,6 +1039,8 @@ void BufferCacheRuntime::BindTransformFeedbackBuffers(VideoCommon::HostBindings< boost::container::static_vector buffer_handles(bindings.buffers.size()); for (u32 i = 0; i < bindings.buffers.size(); ++i) { auto handle = bindings.buffers[i]->Handle(); + NoteCheckUse(handle, bindings.offsets[i], bindings.sizes[i], + UploadReorderCheck::Source::TransformFeedback); if (handle == VK_NULL_HANDLE) { ReserveNullBuffer(); handle = *null_buffer; diff --git a/src/video_core/renderer_vulkan/vk_buffer_cache.h b/src/video_core/renderer_vulkan/vk_buffer_cache.h index 9608d70064..d4b98afed3 100644 --- a/src/video_core/renderer_vulkan/vk_buffer_cache.h +++ b/src/video_core/renderer_vulkan/vk_buffer_cache.h @@ -6,18 +6,23 @@ #pragma once +#include #include +#include +#include #include #include "video_core/buffer_cache/buffer_cache_base.h" #include "video_core/buffer_cache/memory_tracker_base.h" -#include "video_core/buffer_cache/usage_tracker.h" +#include "video_core/buffer_cache/upload_order.h" #include "video_core/engines/maxwell_3d.h" #include "video_core/renderer_vulkan/vk_compute_pass.h" +#include "video_core/renderer_vulkan/nx2_list_slots.h" #include "video_core/renderer_vulkan/vk_multi_range_buffer.h" #include "video_core/renderer_vulkan/vk_staging_buffer_pool.h" #include "video_core/renderer_vulkan/vk_update_descriptor.h" +#include "video_core/renderer_vulkan/vk_upload_reorder_check.h" #include "video_core/surface.h" #include "video_core/vulkan_common/vulkan_memory_allocator.h" #include "video_core/vulkan_common/vulkan_wrapper.h" @@ -55,19 +60,22 @@ class Buffer : public VideoCommon::BufferBase { return buffer.Location(); } - [[nodiscard]] bool IsRegionUsed(u64 offset, u64 size) const noexcept { - return tracker.IsUsed(offset, size); + /// The bytes that work recorded in the current submission uses, and the newest earlier + /// submission whose work used the buffer (BufferCacheRuntime::DecideUploadOrder). + [[nodiscard]] const VideoCommon::SubmissionUses& Uses() const noexcept { + return uses; } - void MarkUsage(u64 offset, u64 size) noexcept; + /// TEST (EDEN_NX2_GPU_COST): the newest tick that used any 4 KiB page of the range, 0 when + /// none did since the trace started. + [[nodiscard]] u64 NewestUseTick(u64 offset, u64 size) const noexcept; - void ResetUsageTracking() noexcept { - tracker.Reset(); - } + /// GPU work recorded now reads or writes [offset, offset + size). + void MarkUsage(u64 offset, u64 size) noexcept; - [[nodiscard]] u64 LastUsageTick() const noexcept { - return last_usage_tick; - } + /// An upload recorded now writes [offset, offset + size): uploads of these bytes after it in + /// this submission run after it (DecideUploadOrder). The check hears of it from the copy. + void TrackUploadWrite(u64 offset, u64 size) noexcept; [[nodiscard]] std::span Mapped() const noexcept { return buffer.IsHostCoherent() ? buffer.Mapped() : std::span{}; @@ -89,11 +97,15 @@ class Buffer : public VideoCommon::BufferBase { Scheduler* scheduler{}; vk::Buffer buffer; std::vector views; - VideoCommon::UsageTracker tracker; + VideoCommon::SubmissionUses uses; VkDeviceAddress device_address{}; - u64 last_usage_tick{}; bool is_null{}; bool sparse_compatible{}; + /// TEST (EDEN_NX2_GPU_COST): the tick of each 4 KiB page's last use, and the last ranges + /// noted with their tick (draws bind the same ranges again and again). + std::vector use_ticks; + std::array, 4> noted_uses{}; + u32 noted_cursor{}; }; class QuadArrayIndexBuffer; @@ -152,6 +164,11 @@ class BufferCacheRuntime { /// to be done. [[nodiscard]] bool IsFreeNow(u64 tick); + /// TEST (EDEN_NX2_READ_TRACE): Scheduler::TraceMark, TraceCounts and TraceCountWork. + u64 TraceMark(); + VideoCommon::ReadTrace::Counts TraceCounts() const; + void TraceCountWork(bool dispatch); + u64 GetDeviceLocalMemory() const; u64 GetDeviceMemoryUsage() const; @@ -169,15 +186,22 @@ class BufferCacheRuntime { [[nodiscard]] StagingBufferRef DownloadStagingBuffer(size_t size, bool deferred = false); - bool CanReorderUpload(const Buffer& buffer, std::span copies); + /// Where an upload of `copies` into buffer runs (VideoCommon::DecideUploadOrder): in order + /// when work recorded in this submission uses a byte it writes, otherwise in the upload + /// command buffer. Tracks the copies as uses of their bytes. + [[nodiscard]] VideoCommon::UploadOrder DecideUploadOrder( + Buffer& buffer, std::span copies); void FreeDeferredStagingBuffer(StagingBufferRef& ref); void PreCopyBarrier(); + /// order: where an upload from the stream buffer runs (DecideUploadOrder); other copies run + /// in order. location: the caller, which the render pass end and the copy are counted at. void CopyBuffer(VkBuffer dst_buffer, VkBuffer src_buffer, std::span copies, bool barrier, - bool can_reorder_upload = false); + VideoCommon::UploadOrder order = {}, + std::source_location location = std::source_location::current()); void PostCopyBarrier(); @@ -228,6 +252,22 @@ class BufferCacheRuntime { return last_stream_uniform; } + /// GPU-written list words the host GPU took over (Nx2ListIndirect), owed in a compute + /// dispatch's constant buffer [device_addr, +size) that `buffer` holds at `offset`, filled + /// already: the slot words go over the placeholders. `read(device)`: guest memory's word. + template + void Nx2PatchComputeUniform(DAddr device_addr, u32 size, VkBuffer buffer, u64 offset, + Read&& read) { + if (nx2_list_slots.HasOwed()) [[unlikely]] { + nx2_list_slots.PatchUniform(device_addr, size, buffer, offset, read); + } + } + + /// The slots of GPU-written list words (Nx2ListIndirect). + [[nodiscard]] Nx2ListSlots& Nx2Slots() noexcept { + return nx2_list_slots; + } + /// Adds `ref` to the descriptor payload exactly as BindMappedUniformBuffer did when it made /// the allocation (it is the very same call). The caller guarantees the allocation is still /// live: made on the scheduler tick that is current now, so the staging pool has not recycled @@ -253,6 +293,7 @@ class BufferCacheRuntime { } void PushMultiRangeSource(const Buffer& buffer, u32 offset, u32 size) { + NoteCheckUse(buffer.Handle(), offset, size, UploadReorderCheck::Source::MultiRange); const vk::MemoryLocation location = buffer.Location(); multi_range_sources.push_back(MultiRangeSource{ .handle = buffer.Handle(), @@ -291,6 +332,7 @@ class BufferCacheRuntime { if (format == VideoCore::Surface::PixelFormat::Invalid) { format = VideoCore::Surface::PixelFormat::R8_UNORM; } + NoteCheckUse(buffer.Handle(), offset, size, UploadReorderCheck::Source::Texel); guest_descriptor_queue.AddTexelBuffer(buffer.View(offset, size, format), buffer.DeviceAddress(), offset, size, TexelBufferFormat(format)); @@ -305,8 +347,16 @@ class BufferCacheRuntime { } private: + /// TEST (EDEN_NX2_UPLOAD_REORDER_CHECK): work recorded now uses these bytes. + void NoteCheckUse(VkBuffer handle, u64 offset, u64 size, UploadReorderCheck::Source source) { + if (reorder_check) [[unlikely]] { + reorder_check->NoteUse(handle, offset, size, source); + } + } + void BindBuffer(const Buffer& buffer, u32 offset, u32 size) { const VkBuffer handle = buffer.Handle(); + NoteCheckUse(handle, offset, size, UploadReorderCheck::Source::Descriptor); if (handle == VK_NULL_HANDLE) { guest_descriptor_queue.AddBuffer(handle, 0, 0, VK_WHOLE_SIZE); } else { @@ -316,6 +366,10 @@ class BufferCacheRuntime { VkFormat TexelBufferFormat(VideoCore::Surface::PixelFormat format) const; + /// TEST (EDEN_NX2_GPU_COST): counts an upload by where it ran and why. + void TraceUpload(const Buffer& buffer, std::span copies, + bool reordered); + void ReserveNullBuffer(); vk::Buffer CreateNullBuffer(); @@ -342,6 +396,9 @@ class BufferCacheRuntime { StreamUniformRef last_stream_uniform{}; + /// TEST: EDEN_NX2_UPLOAD_REORDER_CHECK + std::unique_ptr reorder_check; + bool limit_dynamic_storage_buffers = false; u32 max_dynamic_storage_buffers = (std::numeric_limits::max)(); @@ -351,6 +408,8 @@ class BufferCacheRuntime { VkIndexType bound_index_type = VK_INDEX_TYPE_UINT32; u64 bound_index_epoch = 0; bool bound_index_valid = false; + + Nx2ListSlots nx2_list_slots{memory_allocator, scheduler}; }; struct BufferCacheParams { diff --git a/src/video_core/renderer_vulkan/vk_compute_pass.cpp b/src/video_core/renderer_vulkan/vk_compute_pass.cpp index bf1495f30b..4ae62120ae 100644 --- a/src/video_core/renderer_vulkan/vk_compute_pass.cpp +++ b/src/video_core/renderer_vulkan/vk_compute_pass.cpp @@ -24,6 +24,7 @@ #include "video_core/host_shaders/queries_prefix_scan_sum_comp_spv.h" #include "video_core/host_shaders/queries_prefix_scan_sum_nosubgroups_comp_spv.h" #include "video_core/host_shaders/resolve_conditional_render_comp_spv.h" +#include "video_core/host_shaders/resolve_query_availability_comp_spv.h" #include "video_core/host_shaders/vulkan_quad_indexed_comp_spv.h" #include "video_core/host_shaders/vulkan_uint8_comp_spv.h" #include "video_core/host_shaders/vulkan_vertex_repack_comp_spv.h" @@ -539,6 +540,53 @@ void ConditionalRenderingResolvePass::Resolve(VkBuffer dst_buffer, VkBuffer src_ }); } +QueryAvailabilityResolvePass::QueryAvailabilityResolvePass( + const Device& device_, Scheduler& scheduler_, DescriptorPool& descriptor_pool_, + ComputePassDescriptorQueue& compute_pass_descriptor_queue_) + : ComputePass(device_, scheduler_, descriptor_pool_, INPUT_OUTPUT_DESCRIPTOR_SET_BINDINGS, + INPUT_OUTPUT_DESCRIPTOR_UPDATE_TEMPLATE, INPUT_OUTPUT_BANK_INFO, + COMPUTE_PUSH_CONSTANT_RANGE, RESOLVE_QUERY_AVAILABILITY_COMP_SPV), + scheduler{scheduler_}, compute_pass_descriptor_queue{compute_pass_descriptor_queue_} {} + +void QueryAvailabilityResolvePass::Resolve(VkBuffer dst_buffer, VkDeviceSize dst_offset, + VkBuffer src_buffer, VkDeviceSize src_offset, + u32 count) { + if (count == 0) { + return; + } + compute_pass_descriptor_queue.Acquire(scheduler, 2); + compute_pass_descriptor_queue.AddBuffer(src_buffer, src_offset, VkDeviceSize{count} * 16); + compute_pass_descriptor_queue.AddBuffer(dst_buffer, dst_offset, VkDeviceSize{count} * 8); + const void* const descriptor_data{compute_pass_descriptor_queue.UpdateData()}; + + scheduler.RequestOutsideRenderPassOperationContext(); + scheduler.Record([this, descriptor_data, count](vk::CommandBuffer cmdbuf) { + static constexpr VkMemoryBarrier read_barrier{ + .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, + .dstAccessMask = VK_ACCESS_SHADER_READ_BIT | VK_ACCESS_SHADER_WRITE_BIT, + }; + static constexpr VkMemoryBarrier write_barrier{ + .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_SHADER_WRITE_BIT, + .dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT | VK_ACCESS_SHADER_READ_BIT, + }; + const VkDescriptorSet set = descriptor_allocator.Commit(); + device.GetLogical().UpdateDescriptorSet(set, *descriptor_template, descriptor_data); + cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, + 0, read_barrier); + cmdbuf.BindPipeline(VK_PIPELINE_BIND_POINT_COMPUTE, *pipeline); + cmdbuf.BindDescriptorSets(VK_PIPELINE_BIND_POINT_COMPUTE, *layout, 0, set, {}); + cmdbuf.PushConstants(*layout, VK_SHADER_STAGE_COMPUTE_BIT, count); + cmdbuf.Dispatch((count + 63) / 64, 1, 1); + cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, + VK_PIPELINE_STAGE_TRANSFER_BIT | VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, + 0, write_barrier); + }); +} + QueriesPrefixScanPass::QueriesPrefixScanPass( const Device& device_, Scheduler& scheduler_, DescriptorPool& descriptor_pool_, ComputePassDescriptorQueue& compute_pass_descriptor_queue_) diff --git a/src/video_core/renderer_vulkan/vk_compute_pass.h b/src/video_core/renderer_vulkan/vk_compute_pass.h index a322b0474e..92a5beba40 100644 --- a/src/video_core/renderer_vulkan/vk_compute_pass.h +++ b/src/video_core/renderer_vulkan/vk_compute_pass.h @@ -120,6 +120,25 @@ class ConditionalRenderingResolvePass final : public ComputePass { ComputePassDescriptorQueue& compute_pass_descriptor_queue; }; +/// Resolves samples query results copied with availability (resolve_query_availability.comp): the +/// count when the query had landed, 1024 samples (visible) otherwise. +class QueryAvailabilityResolvePass final : public ComputePass { +public: + explicit QueryAvailabilityResolvePass(const Device& device_, Scheduler& scheduler_, + DescriptorPool& descriptor_pool_, + ComputePassDescriptorQueue& compute_pass_descriptor_queue_); + + /// `count` entries: 16 bytes each at `src_offset` in `src_buffer`, 8 bytes each at + /// `dst_offset` in `dst_buffer`. Orders the copy that wrote the source before it, and its own + /// writes before later transfer and shader reads. + void Resolve(VkBuffer dst_buffer, VkDeviceSize dst_offset, VkBuffer src_buffer, + VkDeviceSize src_offset, u32 count); + +private: + Scheduler& scheduler; + ComputePassDescriptorQueue& compute_pass_descriptor_queue; +}; + class QueriesPrefixScanPass final : public ComputePass { public: explicit QueriesPrefixScanPass(const Device& device_, Scheduler& scheduler_, diff --git a/src/video_core/renderer_vulkan/vk_gpu_cost.cpp b/src/video_core/renderer_vulkan/vk_gpu_cost.cpp new file mode 100644 index 0000000000..95d72f7a4a --- /dev/null +++ b/src/video_core/renderer_vulkan/vk_gpu_cost.cpp @@ -0,0 +1,1031 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "common/dkb_frame_trace.h" +#include "common/logging.h" +#include "video_core/renderer_vulkan/vk_gpu_cost.h" +#include "video_core/renderer_vulkan/vk_scheduler.h" +#include "video_core/vulkan_common/vulkan_device.h" + +namespace Vulkan { +namespace { + +enum class Op : u8 { + SubmitStart, ///< top of pipe at the submission's start: the origin of its intervals + PriorTail, ///< until the work submitted before it is done + MainStart, ///< from the upload command buffer's end to the main one's start + SubmitEnd, ///< from the last command's work to the submission's end + Draw, + DrawIndirect, + Dispatch, + DispatchIndirect, + PassBegin, + PassBeginClear, + PassResume, ///< a begin on the framebuffer of the pass ended last + PassResumeClear, + PassEnd, + Barrier, + ImageBarrier, + LayoutBarrier, ///< an image barrier that changes a layout + WaitEvents, + SetEvent, + CopyBuffer, + CopyImage, + CopyBufferToImage, + CopyImageToBuffer, + BlitImage, + ResolveImage, + ClearColorImage, + ClearAttachments, + FillBuffer, + QueryBegin, + QueryEnd, + QueryReset, + QueryCopy, + ConditionalBegin, + ConditionalEnd, + XfbBegin, + XfbEnd, + Count, +}; +constexpr size_t NUM_OPS = static_cast(Op::Count); + +constexpr std::array OP_NAMES{ + "submission start", "prior work", "main start", "submission end", "draw", "indirect draw", + "dispatch", "indirect dispatch", "begin", "begin+clear", "resume", "resume+clear", "end", + "barrier", "image barrier", "layout change", "wait events", "set event", "copy buffer", + "copy image", "buffer to image", "image to buffer", "blit", "resolve", "clear image", + "clear attachments", "fill buffer", "query begin", "query end", "query reset", "query copy", + "conditional begin", "conditional end", "xfb begin", "xfb end", +}; + +static_assert(OP_NAMES[NUM_OPS - 1] != nullptr); + +constexpr bool IsMarker(Op op) { + return op <= Op::SubmitEnd; +} + +constexpr bool IsPassBegin(Op op) { + return op >= Op::PassBegin && op <= Op::PassResumeClear; +} + +constexpr bool IsBarrier(Op op) { + return op >= Op::Barrier && op <= Op::SetEvent; +} + +/// What ran between a render pass's end and the next begin. +enum Between : u8 { + BetweenBarrier = 1 << 0, + BetweenCopy = 1 << 1, + BetweenDispatch = 1 << 2, + BetweenQuery = 1 << 3, + BetweenClear = 1 << 4, + BetweenOther = 1 << 5, +}; +constexpr size_t NUM_BETWEEN = 6; +constexpr std::array BETWEEN_NAMES{ + "barriers", "copies", "dispatches", "queries", "clears", "other", +}; +static_assert(BETWEEN_NAMES[NUM_BETWEEN - 1] != nullptr); + +constexpr u8 BetweenBit(Op op) { + if (IsBarrier(op)) { + return BetweenBarrier; + } + switch (op) { + case Op::CopyBuffer: + case Op::CopyImage: + case Op::CopyBufferToImage: + case Op::CopyImageToBuffer: + case Op::BlitImage: + case Op::ResolveImage: + return BetweenCopy; + case Op::Dispatch: + case Op::DispatchIndirect: + return BetweenDispatch; + case Op::QueryBegin: + case Op::QueryEnd: + case Op::QueryReset: + case Op::QueryCopy: + return BetweenQuery; + case Op::ClearColorImage: + case Op::FillBuffer: + return BetweenClear; + default: + return BetweenOther; + } +} + +/// The groups of the "GPU cost" line; every timed interval is in exactly one. +enum class Category : u8 { + Draws, + Dispatches, + PassEnds, + PassBegins, + Barriers, + Uploads, + BufferCopies, + ImageUploads, + ImageDownloads, + ImageCopies, + Clears, + Queries, + Other, + Submissions, + Present, + Count, +}; +constexpr size_t NUM_CATEGORIES = static_cast(Category::Count); +constexpr std::array CATEGORY_NAMES{ + "draws", "dispatches", "pass ends (with the barrier after)", + "pass begins", "barriers", "uploads (upload command buffer)", + "buffer copies", "image uploads", "image downloads", + "image copies", "clears", "queries", + "conditional/xfb", "submissions", "present path", +}; +static_assert(CATEGORY_NAMES[NUM_CATEGORIES - 1] != nullptr); + +constexpr Category CategoryOf(RecordSite::Role role, Op op, bool upload, bool present) { + if (present) { + return Category::Present; + } + if (IsMarker(op) || role == RecordSite::Submit) { + return Category::Submissions; + } + if (role == RecordSite::PassEnd || op == Op::PassEnd) { + return Category::PassEnds; + } + if (role == RecordSite::PassBegin || IsPassBegin(op)) { + return Category::PassBegins; + } + if (IsBarrier(op)) { + return Category::Barriers; + } + switch (op) { + case Op::Draw: + case Op::DrawIndirect: + return Category::Draws; + case Op::Dispatch: + case Op::DispatchIndirect: + return Category::Dispatches; + case Op::CopyBuffer: + return upload ? Category::Uploads : Category::BufferCopies; + case Op::CopyBufferToImage: + return Category::ImageUploads; + case Op::CopyImageToBuffer: + return Category::ImageDownloads; + case Op::CopyImage: + case Op::BlitImage: + case Op::ResolveImage: + return Category::ImageCopies; + case Op::ClearColorImage: + case Op::ClearAttachments: + case Op::FillBuffer: + return Category::Clears; + case Op::QueryBegin: + case Op::QueryEnd: + case Op::QueryReset: + case Op::QueryCopy: + return Category::Queries; + default: + return Category::Other; + } +} + +constexpr u32 NO_SITE = ~0u; +constexpr u32 NO_EPISODE = ~0u; +constexpr u32 CAPACITY = 32768; +/// Kept for the submissions' start and end timestamps once the others overflow. +constexpr u32 MARKER_RESERVE = 1024; + +enum class Next : u8 { + None, ///< no begin in the sampled frame after it + Same, ///< the next begin has the framebuffer of the ended pass + Other, ///< another framebuffer +}; + +struct TimedOp { + u32 site; + u32 episode; + u32 bytes; + u16 submission; + Op op; + bool upload; +}; + +/// One render pass end and what came before the next begin. +struct Episode { + u32 site; + u32 draws; + VkFramebuffer framebuffer; + u8 between; + Next next; + bool stores; + bool crossed; ///< a submission ended before the next begin + u64 end_ns; + u64 between_ns; + u64 begin_ns; +}; + +struct SiteInfo { + std::string name; + RecordSite::Role role; + bool present; +}; + +struct Cell { + u64 ns; + u64 count; + u64 bytes; +}; + +struct QueryResult { + u64 value; + u64 available; +}; + +struct BreakTotals { + u64 count; + u64 draws; + u64 stores; + u64 same; + u64 other; + u64 none; + u64 crossed; + u64 only_barriers; + u64 nothing; + std::array between; + u64 end_ns; + u64 between_ns; + u64 begin_ns; +}; + +std::string_view BaseName(std::string_view path) { + if (const size_t slash = path.find_last_of("/\\"); slash != std::string_view::npos) { + path.remove_prefix(slash + 1); + } + return path; +} + +/// "Class::Function" from a compiler's function signature (MSVC's has the return type, the +/// calling convention and the parameters; a lambda's ends in "operator ()"). +std::string ShortFunction(std::string_view signature) { + std::string name; + int depth = 0; + for (const char c : signature) { + if (c == '(' && depth == 0) { + break; + } + if (c == '<') { + ++depth; + } else if (c == '>') { + depth = (std::max)(depth - 1, 0); + } else if (depth == 0) { + name.push_back(c); + } + } + const auto trim = [&name] { + while (!name.empty() && (name.back() == ' ' || name.back() == ':')) { + name.pop_back(); + } + }; + trim(); + if (name.ends_with("operator")) { + name.resize(name.size() - 8); + trim(); + } + if (const size_t space = name.rfind(' '); space != std::string::npos) { + name.erase(0, space + 1); + } + size_t doubled = name.find("::::"); + while (doubled != std::string::npos) { + name.erase(doubled, 2); + doubled = name.find("::::"); + } + if (const size_t last = name.rfind("::"); last != std::string::npos && last > 0) { + if (const size_t before = name.rfind("::", last - 1); before != std::string::npos) { + name.erase(0, before + 2); + } + } + return name; +} + +bool IsPresentPath(std::string_view path) { + return path.find("present/") != std::string_view::npos || + path.find("present\\") != std::string_view::npos || + path.find("vk_blit_screen") != std::string_view::npos || + path.find("renderer_vulkan.cpp") != std::string_view::npos || + path.find("vk_present_manager") != std::string_view::npos; +} + +} // Anonymous namespace + +struct GpuCost::Impl { + Impl(const Device& device_, vk::QueryPool pool_, u32 valid_bits_, double ns_per_tick_) + : device{device_}, real{device_.GetDispatchLoader()}, traced{real}, + pool{std::move(pool_)}, valid_bits{valid_bits_}, ns_per_tick{ns_per_tick_}, + host_reset{device_.IsHostQueryResetSupported()} { + stamps.resize(CAPACITY); + window_start = std::chrono::steady_clock::now(); + } + + void Stamp(VkCommandBuffer cmdbuf, Op op, u64 bytes, u32 episode) { + if (!open) { + return; + } + if (count >= (IsMarker(op) ? CAPACITY : CAPACITY - MARKER_RESERVE)) { + ++frame_overflow; + return; + } + real.vkCmdWriteTimestamp(cmdbuf, + op == Op::SubmitStart ? VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT + : VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, + *pool, count); + stamps[count] = { + .site = current_site, + .episode = episode, + .bytes = static_cast((std::min)(bytes, u64{0xffffffff})), + .submission = submission, + .op = op, + .upload = cmdbuf == upload_handle, + }; + ++count; + } + + /// Any command that does work, after it is recorded. + void Work(VkCommandBuffer cmdbuf, Op op, u64 bytes) { + if (cmdbuf == upload_handle) { + Stamp(cmdbuf, op, bytes, NO_EPISODE); + return; + } + if ((op == Op::Draw || op == Op::DrawIndirect) && in_pass) { + ++pass_draws; + } + if (pending != NO_EPISODE && current_role == RecordSite::Own) { + Episode& episode = episodes[pending]; + episode.between = static_cast(episode.between | BetweenBit(op)); + } + Stamp(cmdbuf, op, bytes, pending); + } + + void PassEnded(VkCommandBuffer cmdbuf) { + if (open) { + pending = static_cast(episodes.size()); + episodes.push_back({ + .site = current_site, + .draws = pass_draws, + .framebuffer = pass_framebuffer, + .between = 0, + .next = Next::None, + .stores = (current_flags & RecordSite::PassHadStores) != 0, + .crossed = false, + .end_ns = 0, + .between_ns = 0, + .begin_ns = 0, + }); + } + in_pass = false; + Stamp(cmdbuf, Op::PassEnd, 0, pending); + } + + void PassBegan(VkCommandBuffer cmdbuf, const VkRenderPassBeginInfo& info) { + const u32 closing = pending; + bool same = false; + if (closing != NO_EPISODE) { + Episode& episode = episodes[closing]; + same = episode.framebuffer == info.framebuffer; + episode.next = same ? Next::Same : Next::Other; + pending = NO_EPISODE; + } + in_pass = true; + pass_framebuffer = info.framebuffer; + pass_draws = 0; + const bool clear = info.clearValueCount != 0; + const Op op = same ? (clear ? Op::PassResumeClear : Op::PassResume) + : (clear ? Op::PassBeginClear : Op::PassBegin); + Stamp(cmdbuf, op, 0, closing); + } + + u32 Intern(const RecordSite& site) { + const auto key = std::make_pair(std::make_pair(site.file, site.line), + static_cast(site.role)); + if (const auto it = site_keys.find(key); it != site_keys.end()) { + return it->second; + } + std::string name = SiteName(site.file, site.line, site.function); + const auto [named, inserted] = + site_names.try_emplace(fmt::format("{}#{}", name, static_cast(site.role)), + static_cast(sites.size())); + if (inserted) { + sites.push_back({ + .name = std::move(name), + .role = site.role, + .present = IsPresentPath(site.file), + }); + } + site_keys.emplace(key, named->second); + return named->second; + } + + void StartFrame() { + count = 0; + submission = 0; + frame_overflow = 0; + episodes.clear(); + pending = NO_EPISODE; + in_pass = false; + frame_open = true; + } + + [[nodiscard]] u64 DeltaNs(u64 begin, u64 end) const { + const u64 ticks = Common::DkbFrameTrace::TimestampDelta(begin, end, valid_bits); + return static_cast(static_cast(ticks) * ns_per_tick); + } + + void Attribute(u32 index, u64 ns) { + const TimedOp& stamp = stamps[index]; + const bool has_site = stamp.site != NO_SITE; + const RecordSite::Role role = has_site ? sites[stamp.site].role : RecordSite::Own; + const bool present = has_site && sites[stamp.site].present; + Cell& cell = cells[static_cast(role)][static_cast(stamp.op)] + [stamp.upload ? 1 : 0][present ? 1 : 0]; + cell.ns += ns; + ++cell.count; + cell.bytes += stamp.bytes; + if (has_site) { + Cell& site_cell = by_site[(u64{stamp.site} << 8) | static_cast(stamp.op)]; + site_cell.ns += ns; + ++site_cell.count; + site_cell.bytes += stamp.bytes; + } + if (stamp.episode == NO_EPISODE) { + return; + } + Episode& episode = episodes[stamp.episode]; + if (stamp.op == Op::PassEnd || role == RecordSite::PassEnd) { + episode.end_ns += ns; + } else if (IsPassBegin(stamp.op) || role == RecordSite::PassBegin) { + episode.begin_ns += ns; + } else { + episode.between_ns += ns; + } + } + + void ReadFrame(); + void Report(); + + const Device& device; + const vk::DeviceDispatch real; + vk::DeviceDispatch traced; + vk::QueryPool pool; + const u32 valid_bits; + const double ns_per_tick; + const bool host_reset; + + // The sampled frame (worker thread). + std::vector stamps; + std::vector episodes; + u32 count{}; + u16 submission{}; + u64 frame_overflow{}; + bool frame_open{}; + bool open{}; ///< in a sampled submission + VkCommandBuffer main_handle{}; + VkCommandBuffer upload_handle{}; + + // The command running. + u32 current_site = NO_SITE; + RecordSite::Role current_role = RecordSite::Own; + u8 current_flags{}; + + // The render pass state the commands left. + bool in_pass{}; + VkFramebuffer pass_framebuffer{}; + u32 pass_draws{}; + u32 pending = NO_EPISODE; ///< the episode of the last end, until the next begin + + // Source lines seen, by pointer and by name (a header's line has a pointer in each TU). + std::vector sites; + std::unordered_map site_names; + struct KeyHash { + size_t operator()(const std::pair, u32>& key) const { + return std::hash{}(key.first.first) ^ + (static_cast(key.first.second) << 8) ^ key.second; + } + }; + std::unordered_map, u32>, u32, KeyHash> site_keys; + + // The 10 s window. + std::chrono::steady_clock::time_point window_start; + std::array, 2>, NUM_OPS>, 4> cells{}; + std::unordered_map by_site; + std::unordered_map breaks; + u64 frames{}; + u64 busy_ns{}; + u64 submissions{}; + u64 stamped{}; + u64 overflow{}; + u64 bad{}; + std::vector results; +}; + +namespace { + +GpuCost::Impl* g_impl{}; + +/// A vkCmd* entry that runs the real one, then notes what it did. +template +struct Hook; + +template +struct Hook { + template + static VKAPI_ATTR void VKAPI_CALL Call(VkCommandBuffer cmdbuf, Args... args) { + GpuCost::Impl& self = *g_impl; + (self.real.*member)(cmdbuf, args...); + self.Work(cmdbuf, op, 0); + } +}; + +template +void Install(vk::DeviceDispatch& traced) { + using Pfn = std::remove_reference_t; + if (traced.*member != nullptr) { + traced.*member = &Hook::template Call; + } +} + +Op BarrierOp(u32 image_count, const VkImageMemoryBarrier* images) { + if (image_count == 0) { + return Op::Barrier; + } + for (u32 i = 0; i < image_count; ++i) { + if (images[i].oldLayout != images[i].newLayout) { + return Op::LayoutBarrier; + } + } + return Op::ImageBarrier; +} + +VKAPI_ATTR void VKAPI_CALL PipelineBarrier(VkCommandBuffer cmdbuf, VkPipelineStageFlags src, + VkPipelineStageFlags dst, VkDependencyFlags flags, + u32 memory_count, const VkMemoryBarrier* memory, + u32 buffer_count, const VkBufferMemoryBarrier* buffers, + u32 image_count, const VkImageMemoryBarrier* images) { + GpuCost::Impl& self = *g_impl; + self.real.vkCmdPipelineBarrier(cmdbuf, src, dst, flags, memory_count, memory, buffer_count, + buffers, image_count, images); + self.Work(cmdbuf, BarrierOp(image_count, images), 0); +} + +VKAPI_ATTR void VKAPI_CALL PipelineBarrier2(VkCommandBuffer cmdbuf, const VkDependencyInfo* info) { + GpuCost::Impl& self = *g_impl; + self.real.vkCmdPipelineBarrier2(cmdbuf, info); + Op op = info->imageMemoryBarrierCount == 0 ? Op::Barrier : Op::ImageBarrier; + for (u32 i = 0; i < info->imageMemoryBarrierCount; ++i) { + if (info->pImageMemoryBarriers[i].oldLayout != info->pImageMemoryBarriers[i].newLayout) { + op = Op::LayoutBarrier; + } + } + self.Work(cmdbuf, op, 0); +} + +VKAPI_ATTR void VKAPI_CALL CopyBuffer(VkCommandBuffer cmdbuf, VkBuffer src, VkBuffer dst, + u32 region_count, const VkBufferCopy* regions) { + GpuCost::Impl& self = *g_impl; + self.real.vkCmdCopyBuffer(cmdbuf, src, dst, region_count, regions); + u64 bytes = 0; + for (u32 i = 0; i < region_count; ++i) { + bytes += regions[i].size; + } + self.Work(cmdbuf, Op::CopyBuffer, bytes); +} + +VKAPI_ATTR void VKAPI_CALL BeginRenderPass(VkCommandBuffer cmdbuf, + const VkRenderPassBeginInfo* info, + VkSubpassContents contents) { + GpuCost::Impl& self = *g_impl; + self.real.vkCmdBeginRenderPass(cmdbuf, info, contents); + self.PassBegan(cmdbuf, *info); +} + +VKAPI_ATTR void VKAPI_CALL EndRenderPass(VkCommandBuffer cmdbuf) { + GpuCost::Impl& self = *g_impl; + self.real.vkCmdEndRenderPass(cmdbuf); + self.PassEnded(cmdbuf); +} + +void InstallHooks(vk::DeviceDispatch& traced) { + using D = vk::DeviceDispatch; + Install<&D::vkCmdDraw, Op::Draw>(traced); + Install<&D::vkCmdDrawIndexed, Op::Draw>(traced); + Install<&D::vkCmdDrawMeshTasksEXT, Op::Draw>(traced); + Install<&D::vkCmdDrawIndirect, Op::DrawIndirect>(traced); + Install<&D::vkCmdDrawIndexedIndirect, Op::DrawIndirect>(traced); + Install<&D::vkCmdDrawIndirectCount, Op::DrawIndirect>(traced); + Install<&D::vkCmdDrawIndexedIndirectCount, Op::DrawIndirect>(traced); + Install<&D::vkCmdDrawIndirectByteCountEXT, Op::DrawIndirect>(traced); + Install<&D::vkCmdDispatch, Op::Dispatch>(traced); + Install<&D::vkCmdDispatchIndirect, Op::DispatchIndirect>(traced); + Install<&D::vkCmdWaitEvents, Op::WaitEvents>(traced); + Install<&D::vkCmdSetEvent, Op::SetEvent>(traced); + Install<&D::vkCmdCopyImage, Op::CopyImage>(traced); + Install<&D::vkCmdCopyBufferToImage, Op::CopyBufferToImage>(traced); + Install<&D::vkCmdCopyImageToBuffer, Op::CopyImageToBuffer>(traced); + Install<&D::vkCmdBlitImage, Op::BlitImage>(traced); + Install<&D::vkCmdResolveImage, Op::ResolveImage>(traced); + Install<&D::vkCmdClearColorImage, Op::ClearColorImage>(traced); + Install<&D::vkCmdClearAttachments, Op::ClearAttachments>(traced); + Install<&D::vkCmdFillBuffer, Op::FillBuffer>(traced); + Install<&D::vkCmdBeginQuery, Op::QueryBegin>(traced); + Install<&D::vkCmdEndQuery, Op::QueryEnd>(traced); + Install<&D::vkCmdResetQueryPool, Op::QueryReset>(traced); + Install<&D::vkCmdCopyQueryPoolResults, Op::QueryCopy>(traced); + Install<&D::vkCmdBeginConditionalRenderingEXT, Op::ConditionalBegin>(traced); + Install<&D::vkCmdEndConditionalRenderingEXT, Op::ConditionalEnd>(traced); + Install<&D::vkCmdBeginTransformFeedbackEXT, Op::XfbBegin>(traced); + Install<&D::vkCmdEndTransformFeedbackEXT, Op::XfbEnd>(traced); + if (traced.vkCmdPipelineBarrier) { + traced.vkCmdPipelineBarrier = &PipelineBarrier; + } + if (traced.vkCmdPipelineBarrier2) { + traced.vkCmdPipelineBarrier2 = &PipelineBarrier2; + } + if (traced.vkCmdCopyBuffer) { + traced.vkCmdCopyBuffer = &CopyBuffer; + } + if (traced.vkCmdBeginRenderPass) { + traced.vkCmdBeginRenderPass = &BeginRenderPass; + } + if (traced.vkCmdEndRenderPass) { + traced.vkCmdEndRenderPass = &EndRenderPass; + } +} + +} // Anonymous namespace + +void GpuCost::Impl::ReadFrame() { + results.assign(count, {}); + const VkResult result = device.GetLogical().GetQueryResults( + *pool, 0, count, results.size() * sizeof(results[0]), results.data(), sizeof(results[0]), + VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WITH_AVAILABILITY_BIT); + if (result != VK_SUCCESS) { + ++bad; + return; + } + std::vector> upload_order(submission); + std::vector> main_order(submission); + for (u32 i = 0; i < count; ++i) { + const TimedOp& stamp = stamps[i]; + if (stamp.submission < submission) { + (stamp.upload ? upload_order : main_order)[stamp.submission].push_back(i); + } + } + for (u16 s = 0; s < submission; ++s) { + std::vector& order = upload_order[s]; + order.insert(order.end(), main_order[s].begin(), main_order[s].end()); + if (order.empty() || stamps[order.front()].op != Op::SubmitStart || + stamps[order.back()].op != Op::SubmitEnd || + std::ranges::any_of(order, [&](u32 i) { return results[i].available == 0; })) { + ++bad; + continue; + } + u64 previous = results[order.front()].value; + u64 total = 0; + for (size_t k = 1; k < order.size(); ++k) { + const u64 value = results[order[k]].value; + u64 ns = DeltaNs(previous, value); + if (ns > u64{1'000'000'000}) { + // Out of order: the interval is lost, the rest stays right. + ++bad; + ns = 0; + } + previous = value; + total += ns; + Attribute(order[k], ns); + } + busy_ns += total; + } + for (const Episode& episode : episodes) { + BreakTotals& totals = breaks[episode.site]; + ++totals.count; + totals.draws += episode.draws; + const auto one = [](bool condition) { return condition ? u64{1} : u64{0}; }; + totals.stores += one(episode.stores); + totals.same += one(episode.next == Next::Same); + totals.other += one(episode.next == Next::Other); + totals.none += one(episode.next == Next::None); + totals.crossed += one(episode.crossed); + totals.only_barriers += one(episode.between == BetweenBarrier); + totals.nothing += one(episode.between == 0); + for (size_t bit = 0; bit < NUM_BETWEEN; ++bit) { + totals.between[bit] += one(((episode.between >> bit) & 1u) != 0); + } + totals.end_ns += episode.end_ns; + totals.between_ns += episode.between_ns; + totals.begin_ns += episode.begin_ns; + } + ++frames; + submissions += submission; + stamped += count; + overflow += frame_overflow; +} + +void GpuCost::Impl::Report() { + if (frames == 0) { + return; + } + const double per_frame = 1.0 / static_cast(frames); + const auto ms = [per_frame](u64 ns) { return static_cast(ns) / 1e6 * per_frame; }; + const auto times = [per_frame](u64 n) { return static_cast(n) * per_frame; }; + + // Totals by category, each with its kinds by time. + struct Group { + Cell total{}; + std::array ops{}; + }; + std::array groups{}; + for (size_t role = 0; role < cells.size(); ++role) { + for (size_t op = 0; op < NUM_OPS; ++op) { + for (size_t upload = 0; upload < 2; ++upload) { + for (size_t present = 0; present < 2; ++present) { + const Cell& cell = cells[role][op][upload][present]; + if (cell.count == 0) { + continue; + } + Group& group = groups[static_cast( + CategoryOf(static_cast(role), static_cast(op), + upload != 0, present != 0))]; + group.total.ns += cell.ns; + group.total.count += cell.count; + group.total.bytes += cell.bytes; + group.ops[op].ns += cell.ns; + group.ops[op].count += cell.count; + } + } + } + } + std::string line; + for (size_t c = 0; c < NUM_CATEGORIES; ++c) { + const Group& group = groups[c]; + if (group.total.count == 0) { + continue; + } + line += fmt::format(" | {} {:.3f} ms {:.1f}x", CATEGORY_NAMES[c], ms(group.total.ns), + times(group.total.count)); + if (group.total.bytes != 0) { + line += fmt::format(" {:.0f} KiB", static_cast(group.total.bytes) / 1024.0 * + per_frame); + } + std::vector kinds; + for (size_t op = 0; op < NUM_OPS; ++op) { + if (group.ops[op].count != 0) { + kinds.push_back(op); + } + } + if (kinds.size() > 1) { + std::ranges::sort(kinds, std::greater<>{}, + [&group](size_t op) { return group.ops[op].ns; }); + line += " ("; + for (size_t k = 0; k < kinds.size(); ++k) { + line += fmt::format("{}{} {:.3f}/{:.1f}x", k == 0 ? "" : ", ", OP_NAMES[kinds[k]], + ms(group.ops[kinds[k]].ns), times(group.ops[kinds[k]].count)); + } + line += ")"; + } + } + const auto seconds = std::chrono::duration(std::chrono::steady_clock::now() - + window_start) + .count(); + LOG_INFO(Render_Vulkan, + "GPU cost in {:.1f} s, {} sampled frames, per frame: busy {:.3f} ms in {:.1f} " + "submissions, {:.0f} timestamps ({} over, {} bad){}", + seconds, frames, ms(busy_ns), times(submissions), times(stamped), overflow, bad, line); + + // By source line and kind. + std::unordered_map named; + for (const auto& [key, cell] : by_site) { + const SiteInfo& site = sites[static_cast(key >> 8)]; + const char* role = site.role == RecordSite::PassEnd ? "[pass end] " + : site.role == RecordSite::PassBegin ? "[next pass] " + : site.role == RecordSite::Submit ? "[submission] " + : ""; + Cell& total = named[fmt::format("{}{} at {}", role, OP_NAMES[key & 0xff], site.name)]; + total.ns += cell.ns; + total.count += cell.count; + total.bytes += cell.bytes; + } + std::vector> sorted(named.begin(), named.end()); + std::ranges::sort(sorted, std::greater<>{}, [](const auto& entry) { return entry.second.ns; }); + if (sorted.size() > 28) { + sorted.resize(28); + } + std::string site_line; + for (const auto& [name, cell] : sorted) { + site_line += fmt::format("{}{:.3f} ms {:.1f}x", site_line.empty() ? "" : " | ", + ms(cell.ns), times(cell.count)); + if (cell.bytes != 0) { + site_line += fmt::format(" {:.0f} KiB", + static_cast(cell.bytes) / 1024.0 * per_frame); + } + site_line += fmt::format(" {}", name); + } + LOG_INFO(Render_Vulkan, "GPU cost by site, per frame: {}", site_line); + + // Render pass breaks by the line that ended the pass. + std::unordered_map by_name; + for (const auto& [site, totals] : breaks) { + BreakTotals& merged = by_name[site == NO_SITE ? std::string{"unknown"} : sites[site].name]; + merged.count += totals.count; + merged.draws += totals.draws; + merged.stores += totals.stores; + merged.same += totals.same; + merged.other += totals.other; + merged.none += totals.none; + merged.crossed += totals.crossed; + merged.only_barriers += totals.only_barriers; + merged.nothing += totals.nothing; + for (size_t bit = 0; bit < NUM_BETWEEN; ++bit) { + merged.between[bit] += totals.between[bit]; + } + merged.end_ns += totals.end_ns; + merged.between_ns += totals.between_ns; + merged.begin_ns += totals.begin_ns; + } + std::vector> ends(by_name.begin(), by_name.end()); + std::ranges::sort(ends, std::greater<>{}, [](const auto& entry) { return entry.second.count; }); + if (ends.size() > 14) { + ends.resize(14); + } + for (const auto& [name, totals] : ends) { + const double n = static_cast(totals.count); + const auto percent = [n](u64 part) { return static_cast(part) * 100.0 / n; }; + std::string between; + for (size_t bit = 0; bit < NUM_BETWEEN; ++bit) { + between += + fmt::format(", {} {:.0f}%", BETWEEN_NAMES[bit], percent(totals.between[bit])); + } + LOG_INFO(Render_Vulkan, + "Render pass breaks at {}: {:.1f} a frame, {:.1f} draws in each pass, storage " + "writes in {:.0f}%; next begin: same framebuffer {:.0f}%, another {:.0f}%, none " + "in the frame {:.0f}%, after a submission {:.0f}%; between: only barriers " + "{:.0f}%, nothing {:.0f}%{}; GPU us per break: end {:.1f}, between {:.1f}, next " + "begin {:.1f}", + name, times(totals.count), static_cast(totals.draws) / n, + percent(totals.stores), percent(totals.same), percent(totals.other), + percent(totals.none), percent(totals.crossed), percent(totals.only_barriers), + percent(totals.nothing), between, static_cast(totals.end_ns) / n / 1e3, + static_cast(totals.between_ns) / n / 1e3, + static_cast(totals.begin_ns) / n / 1e3); + } + + cells = {}; + by_site.clear(); + breaks.clear(); + frames = 0; + busy_ns = 0; + submissions = 0; + stamped = 0; + overflow = 0; + bad = 0; + window_start = std::chrono::steady_clock::now(); +} + +bool GpuCost::Enabled() { + const char* const value = std::getenv("EDEN_NX2_GPU_COST"); + return value != nullptr && *value != '\0' && *value != '0'; +} + +bool GpuCost::TracesUploads() { + const char* const value = std::getenv("EDEN_NX2_GPU_COST"); + return value != nullptr && std::atoi(value) >= 2; +} + +std::unique_ptr GpuCost::Create(const Device& device, u32 timestamp_bits, + float timestamp_period, bool frames_sampled) { + if (!Enabled()) { + return nullptr; + } + if (!frames_sampled) { + LOG_WARNING(Render_Vulkan, "GPU cost (EDEN_NX2_GPU_COST) needs EDEN_DKB_FRAME_TRACE=1 and " + "device timestamps: off"); + return nullptr; + } + vk::QueryPool pool; + try { + pool = device.GetLogical().CreateQueryPool({ + .sType = VK_STRUCTURE_TYPE_QUERY_POOL_CREATE_INFO, + .pNext = nullptr, + .flags = 0, + .queryType = VK_QUERY_TYPE_TIMESTAMP, + .queryCount = CAPACITY, + .pipelineStatistics = 0, + }); + } catch (const vk::Exception& error) { + LOG_WARNING(Render_Vulkan, "GPU cost: no query pool: {}", error.what()); + return nullptr; + } + auto impl = std::make_unique(device, std::move(pool), timestamp_bits, + static_cast(timestamp_period)); + InstallHooks(impl->traced); + if (impl->host_reset) { + device.GetLogical().ResetQueryPool(*impl->pool, 0, CAPACITY); + } + g_impl = impl.get(); + LOG_INFO(Render_Vulkan, + "GPU cost (EDEN_NX2_GPU_COST): timing every Vulkan command that does work in the " + "frames EDEN_DKB_FRAME_TRACE samples, {} queries, {} reset", + CAPACITY, impl->host_reset ? "host" : "command buffer"); + return std::unique_ptr(new GpuCost(std::move(impl))); +} + +GpuCost::GpuCost(std::unique_ptr impl_) : impl{std::move(impl_)} {} + +GpuCost::~GpuCost() { + if (g_impl == impl.get()) { + g_impl = nullptr; + } +} + +std::string GpuCost::SiteName(const char* file, u32 line, const char* function) { + return fmt::format("{}:{} {}", BaseName(file), line, ShortFunction(function)); +} + +void GpuCost::BeginSubmission(vk::CommandBuffer& cmdbuf, vk::CommandBuffer& upload_cmdbuf, + bool first_of_frame) { + Impl& self = *impl; + if (first_of_frame) { + self.StartFrame(); + } + if (!self.frame_open || self.submission == 0xffff) { + return; + } + self.main_handle = *cmdbuf; + self.upload_handle = *upload_cmdbuf; + if (first_of_frame && !self.host_reset) { + self.real.vkCmdResetQueryPool(self.upload_handle, *self.pool, 0, CAPACITY); + } + self.open = true; + self.current_site = NO_SITE; + self.current_role = RecordSite::Own; + self.current_flags = 0; + self.Stamp(self.upload_handle, Op::SubmitStart, 0, NO_EPISODE); + self.Stamp(self.upload_handle, Op::PriorTail, 0, NO_EPISODE); + self.Stamp(self.main_handle, Op::MainStart, 0, NO_EPISODE); + cmdbuf = vk::CommandBuffer(self.main_handle, self.traced); + upload_cmdbuf = vk::CommandBuffer(self.upload_handle, self.traced); +} + +void GpuCost::BeginCommand(const RecordSite& site) { + Impl& self = *impl; + self.current_site = self.Intern(site); + self.current_role = site.role; + self.current_flags = site.flags; +} + +void GpuCost::EndSubmission(vk::CommandBuffer cmdbuf) { + Impl& self = *impl; + if (!self.open) { + return; + } + self.Stamp(*cmdbuf, Op::SubmitEnd, 0, NO_EPISODE); + if (self.pending != NO_EPISODE) { + self.episodes[self.pending].crossed = true; + } + self.open = false; + ++self.submission; + self.upload_handle = VK_NULL_HANDLE; + self.main_handle = VK_NULL_HANDLE; +} + +void GpuCost::FrameDone() { + Impl& self = *impl; + if (!self.frame_open) { + return; + } + self.frame_open = false; + self.open = false; + if (self.count != 0) { + self.ReadFrame(); + if (self.host_reset) { + self.device.GetLogical().ResetQueryPool(*self.pool, 0, CAPACITY); + } + } + if (std::chrono::steady_clock::now() - self.window_start >= std::chrono::seconds{10}) { + self.Report(); + } +} + +} // namespace Vulkan diff --git a/src/video_core/renderer_vulkan/vk_gpu_cost.h b/src/video_core/renderer_vulkan/vk_gpu_cost.h new file mode 100644 index 0000000000..e2effc1c1a --- /dev/null +++ b/src/video_core/renderer_vulkan/vk_gpu_cost.h @@ -0,0 +1,77 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include + +#include "common/common_types.h" +#include "video_core/vulkan_common/vulkan_wrapper.h" + +// TEST (EDEN_NX2_GPU_COST=1, in the frames EDEN_DKB_FRAME_TRACE=1 samples): where the host GPU's +// time goes, the part outside the timed draws and dispatches included. +// +// A sampled submission's command buffers record through a copy of the dispatch table whose +// vkCmd* entries count each call and, after each one that does work (not binds or dynamic +// state), write a bottom-of-pipe timestamp. Consecutive timestamps of a command buffer split the +// submission's GPU time, from a top-of-pipe one at its start to one at its end, into intervals +// that each end with one Vulkan command: the time from the end of the work before it to the end +// of that command's own work. Every interval goes to the command's kind (draw, render pass +// begin or end, barrier, copy, clear, query...) and to the source line that recorded it +// (Scheduler::Record's caller; a render pass's end to the line that ended it, its begin to the +// line that ended the pass before). +// +// Render pass breaks: after each end, what ran before the next begin (only barriers, copies, +// dispatches, queries, clears) and whether that begin resumed the same framebuffer. +// +// This only reads: the commands, their order and their synchronization are unchanged. Logged +// every 10 s over the frames sampled in it, per frame: "GPU cost" (totals by kind), "GPU cost +// by site", and one "Render pass breaks" line for each of the lines that ended the most passes. +// EDEN_NX2_GPU_COST=2 also logs "Uploads": where each upload's pages were last used. +namespace Vulkan { + +class Device; +struct RecordSite; + +class GpuCost { +public: + /// Whether EDEN_NX2_GPU_COST is set. + [[nodiscard]] static bool Enabled(); + + /// EDEN_NX2_GPU_COST=2: also classify the uploads that run in order (vk_buffer_cache.cpp). + /// Its page ticks cost the GPU thread several microseconds per bind of a large buffer. + [[nodiscard]] static bool TracesUploads(); + + /// Null unless EDEN_NX2_GPU_COST is set, frames are sampled (EDEN_DKB_FRAME_TRACE's + /// timestamp pools exist) and the query pool can be created. + static std::unique_ptr Create(const Device& device, u32 timestamp_bits, + float timestamp_period, bool frames_sampled); + ~GpuCost(); + + GpuCost(const GpuCost&) = delete; + GpuCost& operator=(const GpuCost&) = delete; + + /// "file:line Class::Function" of a source location. + [[nodiscard]] static std::string SiteName(const char* file, u32 line, const char* function); + + /// Worker thread: a sampled submission starts in these command buffers. Writes its start + /// timestamps and switches both to the tracing dispatch table. + void BeginSubmission(vk::CommandBuffer& cmdbuf, vk::CommandBuffer& upload_cmdbuf, + bool first_of_frame); + /// Worker thread: the site of the command about to run. + void BeginCommand(const RecordSite& site); + /// Worker thread, in the submission's last command before its command buffers end. + void EndSubmission(vk::CommandBuffer cmdbuf); + /// Worker thread: every submission of the sampled frame is done on the GPU. + void FrameDone(); + + struct Impl; + +private: + explicit GpuCost(std::unique_ptr impl); + + std::unique_ptr impl; +}; + +} // namespace Vulkan diff --git a/src/video_core/renderer_vulkan/vk_graphics_pipeline.cpp b/src/video_core/renderer_vulkan/vk_graphics_pipeline.cpp index 1cbdb85e91..5c64693774 100644 --- a/src/video_core/renderer_vulkan/vk_graphics_pipeline.cpp +++ b/src/video_core/renderer_vulkan/vk_graphics_pipeline.cpp @@ -727,15 +727,18 @@ bool GraphicsPipeline::ConfigureImpl(bool is_indexed) { const auto config_stage{[&](size_t stage) LAMBDA_FORCEINLINE { const Shader::Info& info{stage_infos[stage]}; - buffer_cache.UnbindGraphicsStorageBuffers(stage); - if constexpr (Spec::has_storage_buffers) { - size_t ssbo_index{}; - for (const auto& desc : info.storage_buffers_descriptors) { - ASSERT(desc.count == 1); - buffer_cache.BindGraphicsStorageBuffer(stage, ssbo_index, desc.cbuf_index, - desc.cbuf_offset, desc.is_written, - desc.indirect_offset); - ++ssbo_index; + { + const Profile::FineScope storage_scope{Profile::Zone::StorageBindings}; + buffer_cache.UnbindGraphicsStorageBuffers(stage); + if constexpr (Spec::has_storage_buffers) { + size_t ssbo_index{}; + for (const auto& desc : info.storage_buffers_descriptors) { + ASSERT(desc.count == 1); + buffer_cache.BindGraphicsStorageBuffer(stage, ssbo_index, desc.cbuf_index, + desc.cbuf_offset, desc.is_written, + desc.indirect_offset); + ++ssbo_index; + } } } const auto& cbufs{maxwell3d->state.shader_stages[stage].const_buffers}; @@ -836,6 +839,7 @@ bool GraphicsPipeline::ConfigureImpl(bool is_indexed) { config_stage(4); } } + Profile::AddFine(Profile::FineCounter::TextureHandles, views.size()); ASSERT(views.size() == num_image_elements); ASSERT(samplers.size() == num_textures); if (resolve_in_full) { @@ -1016,6 +1020,7 @@ bool GraphicsPipeline::ConfigureImpl(bool is_indexed) { { Profile::Scope scope{Profile::Zone::UpdateBuffers}; buffer_cache.UpdateGraphicsBuffers(is_indexed); + const Profile::FineScope geometry_scope{Profile::Zone::GeometryBuffers}; buffer_cache.BindHostGeometryBuffers(is_indexed); } @@ -1176,7 +1181,9 @@ bool GraphicsPipeline::ConfigureDelta(bool is_indexed) { if (replay_images) { Profile::Scope scope{Profile::Zone::PushDescriptors}; const auto [first, count] = kept_image_ranges[stage]; - guest_descriptor_queue.AddEntries({kept_image_entries.data() + first, count}); + if (count != 0) { + guest_descriptor_queue.AddEntries({kept_image_entries.data() + first, count}); + } } else { Profile::Scope scope{Profile::Zone::PushDescriptors}; const size_t first_entry = guest_descriptor_queue.UpdateCount(); diff --git a/src/video_core/renderer_vulkan/vk_pipeline_cache.cpp b/src/video_core/renderer_vulkan/vk_pipeline_cache.cpp index e754e76c2a..c20e6ce462 100644 --- a/src/video_core/renderer_vulkan/vk_pipeline_cache.cpp +++ b/src/video_core/renderer_vulkan/vk_pipeline_cache.cpp @@ -2688,32 +2688,11 @@ std::unique_ptr PipelineCache::CreateGraphicsPipeline() { GraphicsEnvironments environments; GetGraphicsEnvironments(environments, graphics_key.unique_hashes); - // Shader cache v2: two or more stages translate at once on the shader jobs' CPU workers while - // this thread waits. Only this thread may flush guest memory, as reading a texture descriptor - // does, so the descriptor pool is flushed here first; with this thread waiting nothing can - // make it dirty again. A pool too large to flush on every miss translates here as before. + // Texture descriptors are read without a flush (ReadTextureInfo), so parallel translation + // never submits through the GPU thread's scheduler or needs a pool-wide flush. if (std::ranges::count_if(graphics_key.unique_hashes, [](u64 hash) { return hash != 0; }) >= 2) { - constexpr u64 MaxFlushedTexturePoolBytes = u64{1} << 20; - const auto& tex_header = maxwell3d->regs.tex_header; - const u64 pool_bytes = (u64{tex_header.limit} + 1) * sizeof(Tegra::Texture::TICEntry); - if (pool_bytes <= MaxFlushedTexturePoolBytes) { - gpu_memory->FlushRegion(tex_header.Address(), pool_bytes); - for (size_t index = 0; index < Maxwell::MaxShaderProgram; ++index) { - if (graphics_key.unique_hashes[index] != 0) { - environments.envs[index].SetTexturePoolFlushed(); - } - } - parallel.emplace(); - } else { - static std::atomic logged{}; - if (!logged.exchange(true, std::memory_order_relaxed)) { - LOG_INFO(Render_Vulkan, - "Texture descriptor pool of {} bytes: pipelines that use it translate " - "on the GPU thread", - pool_bytes); - } - } + parallel.emplace(); } // Native programs emit their SPIR-V in the pipeline's job instead of here diff --git a/src/video_core/renderer_vulkan/vk_query_cache.cpp b/src/video_core/renderer_vulkan/vk_query_cache.cpp index 76d4f129e4..97bdb768e7 100644 --- a/src/video_core/renderer_vulkan/vk_query_cache.cpp +++ b/src/video_core/renderer_vulkan/vk_query_cache.cpp @@ -85,6 +85,26 @@ class SamplesQueryBank : public VideoCommon::BankBase { void Sync(size_t start, size_t size) { const auto& dev = device.GetLogical(); + // AMD's driver: read what is available without blocking; a result that has not landed + // counts as visible (1024 samples), as in the GPU-side resolve (SyncWrites). + if (device.GetDriverID() == VK_DRIVER_ID_AMD_PROPRIETARY) { + std::array, BANK_SIZE> values{}; + const VkResult query_result = dev.GetQueryResults( + *query_pool, static_cast(start), static_cast(size), + sizeof(values[0]) * size, values.data(), sizeof(values[0]), + VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WITH_AVAILABILITY_BIT); + if (query_result == VK_ERROR_DEVICE_LOST) { + device.ReportLoss(); + throw vk::Exception(query_result); + } + if (query_result != VK_SUCCESS && query_result != VK_NOT_READY) { + throw vk::Exception(query_result); + } + for (size_t i = 0; i < size; ++i) { + host_results[start + i] = values[i].second != 0 ? values[i].first : 1024; + } + return; + } const VkResult query_result = dev.GetQueryResults( *query_pool, static_cast(start), static_cast(size), sizeof(u64) * size, &host_results[start], sizeof(u64), VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WAIT_BIT); @@ -146,6 +166,10 @@ class SamplesStreamer : public BaseStreamer { amend_value = 0; accumulation_value = 0; queries_prefix_scan_pass.emplace(device, scheduler, descriptor_pool, compute_pass_descriptor_queue); + if (device.GetDriverID() == VK_DRIVER_ID_AMD_PROPRIETARY) { + availability_pass.emplace(device, scheduler, descriptor_pool, + compute_pass_descriptor_queue); + } const VkBufferCreateInfo buffer_ci = { .sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, @@ -278,63 +302,161 @@ class SamplesStreamer : public BaseStreamer { const bool wait_on_last = driver_id == VK_DRIVER_ID_AMD_PROPRIETARY || driver_id == VK_DRIVER_ID_AMD_OPEN_SOURCE; - ApplyBanksWideOp(pending_sync, [&](SamplesQueryBank* bank, size_t start, - size_t amount) { - size_t bank_id = bank->GetIndex(); - auto& resolve_buffer = buffers[resolve_buffer_index]; - VkQueryPool query_pool = bank->GetInnerPool(); - scheduler.RequestOutsideRenderPassOperationContext(); - scheduler.Record([start, amount, base_offset, query_pool, wait_on_last, - buffer = *resolve_buffer](vk::CommandBuffer cmdbuf) { - const VkBufferMemoryBarrier copy_query_pool_barrier{ - .sType = VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER, - .pNext = nullptr, - .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, - .dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT, - .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, - .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, - .buffer = buffer, - .offset = base_offset, - .size = amount * SamplesQueryBank::QUERY_SIZE, - }; - - if (!wait_on_last || amount == 1) { - cmdbuf.CopyQueryPoolResults( - query_pool, static_cast(start), static_cast(amount), buffer, - static_cast(base_offset), SamplesQueryBank::QUERY_SIZE, - VK_QUERY_RESULT_WAIT_BIT | VK_QUERY_RESULT_64_BIT); - } else { - const size_t last = amount - 1; - const VkDeviceSize last_offset = - base_offset + last * SamplesQueryBank::QUERY_SIZE; - cmdbuf.CopyQueryPoolResults(query_pool, static_cast(start + last), 1, - buffer, last_offset, SamplesQueryBank::QUERY_SIZE, - VK_QUERY_RESULT_WAIT_BIT | VK_QUERY_RESULT_64_BIT); - // Orders the unwaited copy after the waited one. - const VkBufferMemoryBarrier waited_barrier{ + // Copy only the slots of the queries being synced and zero the slots between them. The + // wide copy waits (WAIT_BIT) on every slot of the span; a slot in it no query ended + // stalls the GPU until the driver gives up (~8 ms each on AMD, 10 times a frame in + // Ocarina of Time's Kokiri Forest). + { + std::map>> runs_by_bank; + for (auto q : pending_sync) { + ApplyBankOp(GetQuery(q), [&](SamplesQueryBank* bank, size_t start, size_t amount) { + runs_by_bank[bank->GetIndex()].emplace_back(start, start + amount); + }); + } + for (auto& [bank_id, runs] : runs_by_bank) { + std::ranges::sort(runs); + std::vector> merged; + for (const auto& run : runs) { + if (!merged.empty() && run.first <= merged.back().second) { + merged.back().second = (std::max)(merged.back().second, run.second); + } else { + merged.push_back(run); + } + } + const size_t span_start = merged.front().first; + const size_t span_end = merged.back().second; + VkQueryPool query_pool = bank_pool.GetBank(bank_id).GetInnerPool(); + auto& resolve_buffer = buffers[resolve_buffer_index]; + // AMD's driver: even after a barrier on all prior work, a result can land after + // the copy (with WAIT_BIT each copy waited ~9.6 ms). Copy without waiting, with each + // slot's availability, then resolve each slot on the GPU to its count when it had + // landed and to a "visible" 1024 samples otherwise: a late result reads as an + // occluded object drawn instead of hidden (the scene vanished) or a stall (9 -> 22 + // FPS in Kokiri Forest). The availability word is what decides, not what the copy + // writes for an unavailable query, which the Vulkan spec leaves undefined + // (approach from Savulska's test on #573). Each run still waits on its newest query + // first (#679), so fewer results come in late: steadier frame times in motion on an + // RX 9070 XT. + const bool resolve_availability = availability_pass.has_value(); + const VkDeviceSize span_slots = span_end - span_start; + const VkDeviceSize availability_offset = base_offset * 2; + const VkBuffer availability_buffer = + resolve_availability ? AvailabilityBuffer(availability_offset + span_slots * 16) + : VkBuffer{}; + scheduler.RequestOutsideRenderPassOperationContext(); + scheduler.Record([merged, span_start, span_end, base_offset, query_pool, + buffer = *resolve_buffer, resolve_availability, availability_buffer, + availability_offset, wait_on_last](vk::CommandBuffer cmdbuf) { + constexpr size_t query_size = SamplesQueryBank::QUERY_SIZE; + for (size_t i = 0; i + 1 < merged.size(); ++i) { + const size_t gap_begin = merged[i].second; + const size_t gap_end = merged[i + 1].first; + cmdbuf.FillBuffer(buffer, base_offset + (gap_begin - span_start) * query_size, + (gap_end - gap_begin) * query_size, 0); + if (resolve_availability) { + // Available with a zero count, so the resolve keeps them at zero: the + // prefix scan of multi-slot queries sums every slot, gaps included. + for (size_t slot = gap_begin; slot < gap_end; ++slot) { + const VkDeviceSize offset = + availability_offset + (slot - span_start) * 16; + cmdbuf.FillBuffer(availability_buffer, offset, 8, 0); + cmdbuf.FillBuffer(availability_buffer, offset + 8, 8, 1); + } + } + } + for (const auto& [start, end] : merged) { + if (resolve_availability) { + const size_t last = end - 1; + const VkDeviceSize last_offset = + availability_offset + (last - span_start) * 16; + cmdbuf.CopyQueryPoolResults( + query_pool, static_cast(last), 1, availability_buffer, + last_offset, 16, + VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WITH_AVAILABILITY_BIT | + VK_QUERY_RESULT_WAIT_BIT); + if (last == start) { + continue; + } + // Orders the unwaited copy after the waited one. + const VkBufferMemoryBarrier waited_barrier{ + .sType = VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, + .dstAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .buffer = availability_buffer, + .offset = last_offset, + .size = 16, + }; + cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, + VK_PIPELINE_STAGE_TRANSFER_BIT, 0, + waited_barrier); + cmdbuf.CopyQueryPoolResults( + query_pool, static_cast(start), static_cast(last - start), + availability_buffer, availability_offset + (start - span_start) * 16, + 16, VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WITH_AVAILABILITY_BIT); + } else if (!wait_on_last || end - start == 1) { + cmdbuf.CopyQueryPoolResults( + query_pool, static_cast(start), static_cast(end - start), + buffer, base_offset + (start - span_start) * query_size, query_size, + VK_QUERY_RESULT_WAIT_BIT | VK_QUERY_RESULT_64_BIT); + } else { + // The newest query of the run with the wait, then the others without. + const size_t last = end - 1; + const VkDeviceSize last_offset = + base_offset + (last - span_start) * query_size; + cmdbuf.CopyQueryPoolResults(query_pool, static_cast(last), 1, + buffer, last_offset, query_size, + VK_QUERY_RESULT_WAIT_BIT | + VK_QUERY_RESULT_64_BIT); + // Orders the unwaited copy after the waited one. + const VkBufferMemoryBarrier waited_barrier{ + .sType = VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, + .dstAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .buffer = buffer, + .offset = last_offset, + .size = query_size, + }; + cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, + VK_PIPELINE_STAGE_TRANSFER_BIT, 0, + waited_barrier); + cmdbuf.CopyQueryPoolResults( + query_pool, static_cast(start), static_cast(last - start), + buffer, base_offset + (start - span_start) * query_size, query_size, + VK_QUERY_RESULT_64_BIT); + } + } + if (resolve_availability) { + return; // QueryAvailabilityResolvePass orders the copies before it. + } + const VkBufferMemoryBarrier copy_query_pool_barrier{ .sType = VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER, .pNext = nullptr, .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, - .dstAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT, + .dstAccessMask = VK_ACCESS_TRANSFER_READ_BIT, .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, .buffer = buffer, - .offset = last_offset, - .size = SamplesQueryBank::QUERY_SIZE, + .offset = base_offset, + .size = (span_end - span_start) * query_size, }; cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, - VK_PIPELINE_STAGE_TRANSFER_BIT, 0, waited_barrier); - cmdbuf.CopyQueryPoolResults( - query_pool, static_cast(start), static_cast(last), buffer, - static_cast(base_offset), SamplesQueryBank::QUERY_SIZE, - VK_QUERY_RESULT_64_BIT); + VK_PIPELINE_STAGE_TRANSFER_BIT, 0, + copy_query_pool_barrier); + }); + if (resolve_availability) { + availability_pass->Resolve(*resolve_buffer, base_offset, availability_buffer, + availability_offset, static_cast(span_slots)); } - cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, - VK_PIPELINE_STAGE_TRANSFER_BIT, 0, copy_query_pool_barrier); - }); - offsets[bank_id] = {start, base_offset}; - base_offset += amount * SamplesQueryBank::QUERY_SIZE; - }); + offsets[bank_id] = {span_start, base_offset}; + base_offset += (span_end - span_start) * SamplesQueryBank::QUERY_SIZE; + } + } // Convert queries bool has_multi_queries = false; @@ -671,6 +793,31 @@ class SamplesStreamer : public BaseStreamer { u64 bank_bound_resets{}; ///< Counts CloseCounter dropped once a query outgrew the bank bound. std::mutex flush_guard; std::optional queries_prefix_scan_pass; + /// AMD: results copied with availability (16 bytes a slot) before QueryAvailabilityResolvePass. + std::optional availability_pass; + std::vector availability_buffers; ///< The last is current; older ones stay alive + VkDeviceSize availability_capacity{}; + + /// The current availability buffer, grown to at least `size` bytes. + VkBuffer AvailabilityBuffer(VkDeviceSize size) { + if (size > availability_capacity) { + availability_capacity = (std::max)(size, availability_capacity * 2); + const VkBufferCreateInfo buffer_ci{ + .sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, + .pNext = nullptr, + .flags = 0, + .size = availability_capacity, + .usage = VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT, + .sharingMode = VK_SHARING_MODE_EXCLUSIVE, + .queueFamilyIndexCount = 0, + .pQueueFamilyIndices = nullptr, + }; + // Commands already recorded may still use the old one: it is kept, not destroyed. + availability_buffers.push_back( + memory_allocator.CreateBuffer(buffer_ci, MemoryUsage::DeviceLocal)); + } + return *availability_buffers.back(); + } }; // Transform feedback queries @@ -1419,6 +1566,9 @@ struct QueryCacheRuntimeImpl { VkConditionalRenderingBeginInfoEXT hcr_setup; VkBuffer hcr_buffer; size_t hcr_offset; + /// The guest bytes hcr_setup reads when they are a guest buffer's, 0 when it reads the + /// resolve buffer. Each resume reads them again, in the submission it is recorded in. + DAddr hcr_guest_address{}; bool hcr_is_set; bool is_hcr_running; @@ -1449,6 +1599,7 @@ void QueryCacheRuntime::EndHostConditionalRendering() { impl->is_hcr_running = false; impl->hcr_buffer = VkBuffer{}; impl->hcr_offset = 0; + impl->hcr_guest_address = 0; } void QueryCacheRuntime::PauseHostConditionalRendering() { @@ -1467,6 +1618,11 @@ void QueryCacheRuntime::ResumeHostConditionalRendering() { return; } if (!impl->is_hcr_running) { + if (impl->hcr_guest_address != 0) { + // An upload of the predicate later in this submission must run after this read. + std::scoped_lock lk(impl->buffer_cache.mutex); + impl->buffer_cache.MarkUsed(impl->hcr_guest_address, 8); + } impl->scheduler.Record([hcr_setup = impl->hcr_setup](vk::CommandBuffer cmdbuf) { cmdbuf.BeginConditionalRenderingEXT(hcr_setup); }); @@ -1497,6 +1653,7 @@ void QueryCacheRuntime::HostConditionalRenderingCompareValueImpl(VideoCommon::Lo } impl->hcr_setup.buffer = impl->hcr_buffer; impl->hcr_setup.offset = impl->hcr_offset; + impl->hcr_guest_address = object.address; impl->hcr_setup.flags = is_equal ? VK_CONDITIONAL_RENDERING_INVERTED_BIT_EXT : 0; impl->hcr_is_set = true; impl->is_hcr_running = false; @@ -1526,6 +1683,7 @@ void QueryCacheRuntime::HostConditionalRenderingCompareBCImpl(DAddr address, boo impl->conditional_resolve_pass->Resolve(*impl->hcr_resolve_buffer, to_resolve, to_resolve_offset, compare_to_zero); impl->hcr_setup.buffer = *impl->hcr_resolve_buffer; + impl->hcr_guest_address = 0; impl->hcr_setup.offset = 0; impl->hcr_setup.flags = is_equal ? 0 : VK_CONDITIONAL_RENDERING_INVERTED_BIT_EXT; impl->hcr_is_set = true; diff --git a/src/video_core/renderer_vulkan/vk_rasterizer.cpp b/src/video_core/renderer_vulkan/vk_rasterizer.cpp index f8aa71f0e1..df4af3f835 100644 --- a/src/video_core/renderer_vulkan/vk_rasterizer.cpp +++ b/src/video_core/renderer_vulkan/vk_rasterizer.cpp @@ -6,6 +6,7 @@ #include #include +#include #include #include #include @@ -35,6 +36,7 @@ #include "common/settings.h" #include "video_core/buffer_cache/buffer_cache.h" #include "video_core/gpu_logging/gpu_logging.h" +#include "video_core/gpu_wait_stats.h" #include "video_core/control/channel_state.h" #include "video_core/native_query_trace.h" #include "video_core/native_indirect_reset.h" @@ -74,7 +76,9 @@ #include "video_core/memory_manager.h" #include "video_core/nx2.h" #include "video_core/nx2_debug_control.h" +#include "video_core/nx2_drain_check.h" #include "video_core/nx2_fast_draw.h" +#include "video_core/nx2_list_stats.h" #include "video_core/nx2_profile.h" namespace Vulkan { @@ -262,6 +266,20 @@ bool IsPrimitiveRestartSupported(const Device& device, VkPrimitiveTopology topol (topology == VK_PRIMITIVE_TOPOLOGY_PATCH_LIST && device.IsPatchListPrimitiveRestartSupported())); } + +/// Before a dispatch: earlier writes visible to it, its indirect arguments included. +void RecordDispatchBarrier(vk::CommandBuffer cmdbuf) { + static constexpr VkMemoryBarrier READ_BARRIER{ + .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, + .pNext = nullptr, + .srcAccessMask = VK_ACCESS_MEMORY_WRITE_BIT, + .dstAccessMask = VK_ACCESS_MEMORY_READ_BIT, + }; + cmdbuf.PipelineBarrier(vk::PIPELINE_STAGE_GRAPHICS_COMPUTE_TRANSFER, + VK_PIPELINE_STAGE_DRAW_INDIRECT_BIT | + VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, + 0, READ_BARRIER); +} } // Anonymous namespace void RasterizerVulkan::ReadMeshEmulationProbe() { @@ -758,6 +776,8 @@ void RasterizerVulkan::PrepareDraw(bool is_indexed, Func&& draw_func, bool is_in } SCOPE_EXIT { + const VideoCommon::Nx2Profile::FineScope tick_scope{ + VideoCommon::Nx2Profile::Zone::TickWork}; gpu.TickWork(); }; bool submitted = false; @@ -843,6 +863,9 @@ void RasterizerVulkan::PrepareDraw(bool is_indexed, Func&& draw_func, bool is_in } return; } + // TEST: up to Configure (EDEN_NX2_GPU_PROFILE=2). + std::optional setup_scope; + setup_scope.emplace(Profile::Zone::DrawSetup); if (VideoCommon::OotNetwork::BypassEnabled(Settings::GetCurrentProgramID())) { TrackOotTonemappedImage(*pipeline); } @@ -1017,6 +1040,7 @@ void RasterizerVulkan::PrepareDraw(bool is_indexed, Func&& draw_func, bool is_in } } } + setup_scope.reset(); const bool delta_holds = delta_pipeline != nullptr && DeltaStillHolds(draw_serial, draw_map_serial); const DrawPath path = delta_holds && !verify_pipeline ? DrawPath::Delta : DrawPath::Full; @@ -1095,9 +1119,13 @@ void RasterizerVulkan::PrepareDraw(bool is_indexed, Func&& draw_func, bool is_in UpdateDynamicStates(pipeline); } - query_cache.NotifySegment(true); - HandleTransformFeedback(); - query_cache.CounterEnable(VideoCommon::QueryType::ZPassPixelCount64, maxwell3d->regs.zpass_pixel_count_enable); + { + const Profile::FineScope queries_scope{Profile::Zone::DrawQueries}; + query_cache.NotifySegment(true); + HandleTransformFeedback(); + query_cache.CounterEnable(VideoCommon::QueryType::ZPassPixelCount64, + maxwell3d->regs.zpass_pixel_count_enable); + } { Profile::Scope scope{Profile::Zone::DrawRecord}; scheduler.TraceGraphicsPipeline(pipeline->ShaderHashes()); @@ -2057,6 +2085,13 @@ void RasterizerVulkan::Clear(u32 layer_count) { } void RasterizerVulkan::DispatchCompute() { + // As after a draw: run the sync operations threads wait for, such as a guest read's flush, + // rather than only at the next draw. + SCOPE_EXIT { + const VideoCommon::Nx2Profile::FineScope tick_scope{ + VideoCommon::Nx2Profile::Zone::TickWork}; + gpu.TickWork(); + }; VideoCommon::Nx2Profile::Scope profile_scope{VideoCommon::Nx2Profile::Zone::Compute}; FlushWork(); gpu_memory->FlushCaching(); @@ -2073,6 +2108,8 @@ void RasterizerVulkan::DispatchCompute() { } std::scoped_lock lock{texture_cache.mutex, buffer_cache.mutex}; std::optional byte_addressed_shadow; + const Tegra::Nx2ListStats::WriterScope list_writer{Tegra::Nx2ListStats::Writer::Dispatch, + pipeline->ShaderHash()}; buffer_cache.Nx2CollectListCopies(true); SCOPE_EXIT { buffer_cache.Nx2CollectListCopies(false); @@ -2081,15 +2118,16 @@ void RasterizerVulkan::DispatchCompute() { } }; const auto indirect_address = kepler_compute->GetIndirectComputeAddress(); + const auto list_slot = kepler_compute->GetIndirectListSlot(); // The full-size 1080p pass is replaced only by FSR (native AA); DLAA through it is untested. const bool can_replace = OcarinaTemporalEnabled() && pipeline->CanReplaceOcarinaUpscale() && (pipeline->ShaderHash() != ComputePipeline::OcarinaFullSizeShader || oot_backend == VideoCommon::OcarinaUpscaleBackend::Fsr) && - !indirect_address && !frame_dump.Capturing(); + !indirect_address && !list_slot && !frame_dump.Capturing(); // Until a dispatch overwrites the scene, each direct one binds only after the effects are - // marked. The indirect path records its bind at once. + // marked. The indirect paths record their bind at once. const bool marking_effects = oot_effects_frame == render_trace_frame && OcarinaTemporalEnabled() && - !indirect_address && !frame_dump.Capturing(); + !indirect_address && !list_slot && !frame_dump.Capturing(); ComputePipeline::BindCommand deferred_bind; if (!pipeline->Configure(*kepler_compute, *gpu_memory, scheduler, buffer_cache, texture_cache, texture_cache_runtime, byte_addressed_shadow, @@ -2135,23 +2173,6 @@ void RasterizerVulkan::DispatchCompute() { const u32 index = Tegra::Texture::TexturePair(handle, qmd.linked_tsc != 0).first; texture_cache.PublishComputeCollisionImage(index); }; - // Earlier writes must be visible to the dispatch, its indirect arguments included, and what it - // writes to the next render pass. - const auto record_read_barrier = [this] { - static constexpr VkMemoryBarrier READ_BARRIER{ - .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, - .pNext = nullptr, - .srcAccessMask = VK_ACCESS_MEMORY_WRITE_BIT, - .dstAccessMask = VK_ACCESS_MEMORY_READ_BIT, - }; - scheduler.Record([](vk::CommandBuffer cmdbuf) { - cmdbuf.PipelineBarrier(vk::PIPELINE_STAGE_GRAPHICS_COMPUTE_TRANSFER, - VK_PIPELINE_STAGE_DRAW_INDIRECT_BIT | - VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, - 0, READ_BARRIER); - }); - scheduler.NoteComputeWrites(); - }; if (frame_dump.CapturingBuffers()) { std::optional> indirect_grid; if (indirect_address) { @@ -2164,6 +2185,43 @@ void RasterizerVulkan::DispatchCompute() { {qmd.block_dim_x, qmd.block_dim_y, qmd.block_dim_z}, indirect_address, indirect_grid); } + if (list_slot) { + // The grid is GPU-written list words (Nx2ListIndirect): the host GPU copied them into the + // slot at the list's place, and dispatches from there. + auto& slots = buffer_cache_runtime.Nx2Slots(); + if (!slots.Held(*list_slot)) [[unlikely]] { + LOG_ERROR(Render_Vulkan, "Nx2 list slot {} dispatched without a hold", *list_slot); + return; + } + const auto [args_buffer, args_offset] = slots.DispatchArguments(*list_slot); + scheduler.RequestOutsideRenderPassOperationContext(); + scheduler.NoteComputeWrites(); + scheduler.TraceComputePipeline(pipeline->ShaderHash()); + scheduler.Record([this, pipeline, byte_addressed_shadow, indirect_buffer = args_buffer, + indirect_offset = args_offset](vk::CommandBuffer cmdbuf) { + RecordDispatchBarrier(cmdbuf); + if (!pipeline->IsBound()) { + return; + } + if (byte_addressed_shadow) { + TextureCacheRuntime::CopyByteAddressedShadow(cmdbuf, *byte_addressed_shadow, false); + } + scheduler.TraceGpuWork(cmdbuf, pipeline->ShaderHash(), 0, {}, [&] { + cmdbuf.DispatchIndirect(indirect_buffer, indirect_offset); + }); + if (byte_addressed_shadow) { + TextureCacheRuntime::CopyByteAddressedShadow(cmdbuf, *byte_addressed_shadow, true); + } + }); + Tegra::Nx2ListIndirect::Stats::Count(Tegra::Nx2ListIndirect::Stats::Get().slot_dispatches); + if (Tegra::Nx2ListIndirect::Stats::g_check) [[unlikely]] { + slots.CheckDispatch(*list_slot); // TEST + } + buffer_cache.NoteWorkRecorded(true); + SubmitNx2ListCopies(); + TryOcarinaUpscale(*pipeline, false); // As after a direct dispatch + return; + } if (indirect_address) { // DispatchIndirect static constexpr auto sync_info = VideoCommon::ObtainBufferSynchronize::FullSynchronize; @@ -2171,10 +2229,11 @@ void RasterizerVulkan::DispatchCompute() { const auto [buffer, offset] = buffer_cache.ObtainBuffer(*indirect_address, 12, sync_info, post_op); scheduler.RequestOutsideRenderPassOperationContext(); - record_read_barrier(); + scheduler.NoteComputeWrites(); scheduler.TraceComputePipeline(pipeline->ShaderHash()); scheduler.Record([this, pipeline, indirect_buffer = buffer->Handle(), indirect_offset = offset](vk::CommandBuffer cmdbuf) { + RecordDispatchBarrier(cmdbuf); if (!pipeline->IsBound()) { return; } @@ -2193,7 +2252,13 @@ void RasterizerVulkan::DispatchCompute() { return; } scheduler.RequestOutsideRenderPassOperationContext(); - record_read_barrier(); + scheduler.NoteComputeWrites(); + // The upscaler's evaluation or the effects mark pass may be recorded next, before the + // dispatch: the barrier goes first. + const bool barrier_first = can_replace || marking_effects; + if (barrier_first) { + scheduler.Record([](vk::CommandBuffer cmdbuf) { RecordDispatchBarrier(cmdbuf); }); + } if (marking_effects) { const auto written = pipeline->WrittenImages(); if (std::ranges::any_of(written, [this](VideoCommon::ImageViewId id) { @@ -2211,7 +2276,10 @@ void RasterizerVulkan::DispatchCompute() { scheduler.TraceComputePipeline(pipeline->ShaderHash()); } scheduler.Record([this, pipeline, dim, byte_addressed_shadow, bind = std::move(deferred_bind), - replacement](vk::CommandBuffer cmdbuf) { + barrier_recorded = barrier_first, replacement](vk::CommandBuffer cmdbuf) { + if (!barrier_recorded) { + RecordDispatchBarrier(cmdbuf); + } if (replacement && !replacement->EvaluationFailed()) { return; } @@ -2271,6 +2339,12 @@ void RasterizerVulkan::ResetCounter(VideoCommon::QueryType type) { void RasterizerVulkan::Query(GPUVAddr gpu_addr, VideoCommon::QueryType type, VideoCommon::QueryPropertiesFlags flags, u32 payload, u32 subreport) { + if (VideoCommon::Nx2DrainCheck::g_enabled && type == VideoCommon::QueryType::Payload) + [[unlikely]] { + if (const auto released = gpu_memory->GpuToCpuAddress(gpu_addr)) { + VideoCommon::Nx2DrainCheck::NoteRelease(*released); + } + } query_cache.CounterReport(gpu_addr, type, flags, payload, subreport); } @@ -2327,6 +2401,56 @@ void RasterizerVulkan::NoteGpuWrittenCommandListSynced(DAddr page) { buffer_cache.Nx2NoteListSynced(page); } +bool RasterizerVulkan::Nx2ListSlotsSupported() const { + // A direct dispatch skips a grid over the device's limits; an indirect one must not get one. + // The slot's height and depth are 16 bits, and a width over 31 bits is no culling result. + const std::array max_dim{device.GetMaxComputeWorkGroupCount()}; + const bool fits = max_dim[0] >= 0x7fffffffU && max_dim[1] >= 0xffffU && max_dim[2] >= 0xffffU; + if (static std::atomic logged{}; !fits && !logged.exchange(true)) { + LOG_WARNING(Render_Vulkan, + "Nx2 list indirect off: the device dispatches at most {}x{}x{} groups", + max_dim[0], max_dim[1], max_dim[2]); + } + return Tegra::Nx2ListIndirect::g_enabled && fits; +} + +std::optional RasterizerVulkan::Nx2AcquireListSlot() { + return buffer_cache_runtime.Nx2Slots().Acquire(); +} + +void RasterizerVulkan::Nx2ReleaseListSlot(u32 slot) { + buffer_cache_runtime.Nx2Slots().Release(slot); +} + +void RasterizerVulkan::Nx2SnapshotListWords(DAddr device_addr, u32 count, u32 slot) { + // The words as GPU work sent before the list left them: the buffer cache's, its CPU writes + // applied (as a dispatch reading them finds them). + gpu_memory->FlushCaching(); + std::scoped_lock lock{buffer_cache.mutex}; + const auto [buffer, offset] = + buffer_cache.ObtainCPUBuffer(device_addr, count * static_cast(sizeof(u32)), + VideoCommon::ObtainBufferSynchronize::FullSynchronize, + VideoCommon::ObtainBufferOperation::DoNothing); + buffer_cache_runtime.Nx2Slots().Snapshot(buffer->Handle(), offset, count, slot); +} + +void RasterizerVulkan::Nx2NoteOwedListWord(DAddr device_addr, u32 value) { + buffer_cache_runtime.Nx2Slots().NoteOwed(device_addr, value, + device_memory.Read(device_addr)); +} + +void RasterizerVulkan::Nx2NoteInlineQmdGrid(DAddr device_addr, u32 x, u32 y, u32 z) { + buffer_cache_runtime.Nx2Slots().NoteInlineQmdGrid(device_addr, x, y, z); +} + +void RasterizerVulkan::Nx2ReadListSlots(std::span slots, std::span words) { + buffer_cache_runtime.Nx2Slots().Read(slots, words); +} + +void RasterizerVulkan::Nx2CheckListSlot(u32 slot, std::span words) { + buffer_cache_runtime.Nx2Slots().CheckSnapshot(slot, words); // TEST +} + void RasterizerVulkan::FlushGpuWrittenCommandListPages(std::span pages) { gpu.InvalidateGPUCache(); std::scoped_lock lock{buffer_cache.mutex}; @@ -2379,6 +2503,11 @@ void RasterizerVulkan::FinishRead(DAddr addr, u64 size) { buffer_cache.FinishReadDownloads(addr, size); } +void RasterizerVulkan::TraceRead(VideoCommon::ReadTrace::Read& read, + VideoCommon::ReadTrace::Step step) { + scheduler.TraceRead(read, step); +} + bool RasterizerVulkan::NoteReportPending(DAddr addr, u64 size) { std::scoped_lock lock{buffer_cache.mutex}; return buffer_cache.NoteReportPending(addr, size); @@ -2431,12 +2560,14 @@ VideoCore::RasterizerDownloadArea RasterizerVulkan::GetFlushArea(DAddr addr, u64 std::scoped_lock lock{buffer_cache.mutex}; buffer_cache.ForgetReadTraps(texture_area->start_address, texture_area->end_address - texture_area->start_address); + texture_area->source = VideoCore::DownloadAreaSource::TextureCache; return *texture_area; } if (buffer_cache.FlushesOnCpuRead()) { std::scoped_lock lock{buffer_cache.mutex}; auto area = buffer_cache.GetFlushArea(addr, size); if (area) { + area->source = VideoCore::DownloadAreaSource::BufferCache; return *area; } } @@ -2453,8 +2584,6 @@ void RasterizerVulkan::PrepareRegionWrite(DAddr addr, u64 size, VideoCommon::Cac return; } std::scoped_lock lock{buffer_cache.mutex}; - // MemoryManager landed the read downloads here just before (FinishRead) and writes guest memory - // next. Landing a newer one would wait for the host GPU with the lock held, from any thread. buffer_cache.ClearReplacedDownload(addr, size); } @@ -2627,6 +2756,7 @@ void RasterizerVulkan::FlushQueryResults(DAddr addr, u64 size) { // Below High GPU accuracy a fence runs its own callback at once, so wait for an operation // queued behind the pending report writes instead. On the split's back the fence's // submission may be deferred, and the fencing thread would wait for this thread to submit it. + const u64 wait_start = VideoCommon::GpuWaitStats::Now(); std::mutex wait_mutex; std::condition_variable wait_cv; bool written{}; @@ -2639,6 +2769,7 @@ void RasterizerVulkan::FlushQueryResults(DAddr addr, u64 size) { scheduler.FlushDeferred(); std::unique_lock lock{wait_mutex}; wait_cv.wait(lock, [&written] { return written; }); + VideoCommon::GpuWaitStats::AddQueryWait(VideoCommon::GpuWaitStats::Now() - wait_start); } void RasterizerVulkan::FlushAndInvalidateRegion(DAddr addr, u64 size, @@ -3505,7 +3636,13 @@ void RasterizerVulkan::FlushWork() { } return std::nullopt; }(); - budget = configured_budget.value_or(SubmissionPolicy::DefaultNativeBudget); + // AMD's driver pays much more per submission. Ocarina of Time on an RX 9070 XT: every 256 + // draws instead of 128 gave +10-12 FPS where it was already fast, every 512 a further + // 35 -> 50 FPS at one spot standing still. + const u32 default_budget = device.GetDriverID() == VK_DRIVER_ID_AMD_PROPRIETARY + ? SubmissionPolicy::AmdNativeBudget + : SubmissionPolicy::DefaultNativeBudget; + budget = configured_budget.value_or(default_budget); } u32 legacy_budget = 4096; if (!native && !android) { @@ -3544,6 +3681,8 @@ bool AccelerateDMA::BufferClear(GPUVAddr src_address, u64 amount, u32 value) { bool AccelerateDMA::BufferCopy(GPUVAddr src_address, GPUVAddr dest_address, u64 amount) { std::scoped_lock lock{buffer_cache.mutex}; + const Tegra::Nx2ListStats::WriterScope list_writer{Tegra::Nx2ListStats::Writer::Copy, + src_address}; buffer_cache.Nx2CollectListCopies(true); const bool copied = buffer_cache.DMACopy(src_address, dest_address, amount); if (buffer_cache.Nx2CopyListPages(scheduler.CurrentTick())) { diff --git a/src/video_core/renderer_vulkan/vk_rasterizer.h b/src/video_core/renderer_vulkan/vk_rasterizer.h index 79d296d51b..4cb4238368 100644 --- a/src/video_core/renderer_vulkan/vk_rasterizer.h +++ b/src/video_core/renderer_vulkan/vk_rasterizer.h @@ -108,6 +108,7 @@ class RasterizerVulkan final : public VideoCore::RasterizerInterface, VideoCommon::CacheType which = VideoCommon::CacheType::All) override; void FlushRegionForRead(DAddr addr, u64 size) override; void FinishRead(DAddr addr, u64 size) override; + void TraceRead(VideoCommon::ReadTrace::Read& read, VideoCommon::ReadTrace::Step step) override; bool NoteReportPending(DAddr addr, u64 size) override; void FinishReport(DAddr addr, u64 size) override; GpuWrittenList ReadGpuWrittenCommandList(DAddr addr, std::span out, bool past_sync, @@ -115,6 +116,14 @@ class RasterizerVulkan final : public VideoCore::RasterizerInterface, bool IsGpuTickDone(u64 tick) override; void NoteGpuWrittenCommandListSynced(DAddr page) override; void FlushGpuWrittenCommandListPages(std::span pages) override; + bool Nx2ListSlotsSupported() const override; + std::optional Nx2AcquireListSlot() override; + void Nx2ReleaseListSlot(u32 slot) override; + void Nx2SnapshotListWords(DAddr device_addr, u32 count, u32 slot) override; + void Nx2NoteOwedListWord(DAddr device_addr, u32 value) override; + void Nx2NoteInlineQmdGrid(DAddr device_addr, u32 x, u32 y, u32 z) override; + void Nx2ReadListSlots(std::span slots, std::span words) override; + void Nx2CheckListSlot(u32 slot, std::span words) override; // TEST bool MustFlushRegion(DAddr addr, u64 size, VideoCommon::CacheType which = VideoCommon::CacheType::All) override; VideoCore::RasterizerDownloadArea GetFlushArea(DAddr addr, u64 size) override; diff --git a/src/video_core/renderer_vulkan/vk_scheduler.cpp b/src/video_core/renderer_vulkan/vk_scheduler.cpp index 7c3241dcf7..7614d1d915 100644 --- a/src/video_core/renderer_vulkan/vk_scheduler.cpp +++ b/src/video_core/renderer_vulkan/vk_scheduler.cpp @@ -4,14 +4,18 @@ // SPDX-FileCopyrightText: Copyright 2019 yuzu Emulator Project // SPDX-License-Identifier: GPL-2.0-or-later +#include +#include #include #include #include #include +#include #include #include #include #include +#include #include #include #include @@ -25,13 +29,16 @@ #include "common/dkb_cpu_trace.h" #include "common/thread.h" #include "video_core/gpu_logging/gpu_logging.h" +#include "video_core/gpu_wait_stats.h" #include "video_core/nx2_profile.h" #include "video_core/renderer_vulkan/vk_command_pool.h" +#include "video_core/renderer_vulkan/vk_gpu_cost.h" #include "video_core/renderer_vulkan/vk_graphics_pipeline.h" #include "video_core/renderer_vulkan/vk_master_semaphore.h" #include "video_core/renderer_vulkan/vk_scheduler.h" #include "video_core/renderer_vulkan/vk_state_tracker.h" #include "video_core/renderer_vulkan/vk_texture_cache.h" +#include "video_core/renderer_vulkan/upload_batch.h" #include "video_core/vulkan_common/vulkan_device.h" #include "video_core/vulkan_common/vulkan_wrapper.h" @@ -44,18 +51,14 @@ std::atomic g_profile_finishes{}; std::atomic g_profile_finish_ns{}; std::atomic g_profile_window_start{}; std::mutex g_profile_end_mutex; -std::map g_profile_ends; +std::map, u64> g_profile_ends; -void CountRenderPassEnd(const char* file, u32 line) { +void CountRenderPassEnd(const std::source_location& location) { if (!VideoCommon::Nx2Profile::Enabled()) { return; } - std::string_view name{file}; - if (const auto slash = name.find_last_of('/'); slash != std::string_view::npos) { - name.remove_prefix(slash + 1); - } std::scoped_lock lock{g_profile_end_mutex}; - ++g_profile_ends[fmt::format("{}:{}", name, line)]; + ++g_profile_ends[{location.file_name(), location.line(), location.function_name()}]; } void ReportSubmitCounts() { @@ -77,14 +80,19 @@ void ReportSubmitCounts() { static_cast(g_profile_renderpasses.exchange(0)) / seconds, static_cast(g_profile_finishes.exchange(0)) / seconds, static_cast(g_profile_finish_ns.exchange(0)) / 1e6 / seconds); - std::vector> ends; + std::map named; { std::scoped_lock lock{g_profile_end_mutex}; for (const auto& [site, count] : g_profile_ends) { - ends.emplace_back(count, site); + const auto& [file, line, function] = site; + named[GpuCost::SiteName(file, line, function)] += count; } g_profile_ends.clear(); } + std::vector> ends; + for (const auto& [site, count] : named) { + ends.emplace_back(count, site); + } std::sort(ends.rbegin(), ends.rend()); for (size_t i = 0; i < ends.size() && i < 16; ++i) { LOG_INFO(Render_Vulkan, "Render pass ends {:.0f}/s at {}", @@ -94,14 +102,18 @@ void ReportSubmitCounts() { } void Scheduler::CommandChunk::ExecuteAll(vk::CommandBuffer cmdbuf, - vk::CommandBuffer upload_cmdbuf) { + vk::CommandBuffer upload_cmdbuf, GpuCost* cost) { auto command = first; - while (command != nullptr) { + for (size_t index = 0; command != nullptr; ++index) { auto next = command->GetNext(); + if (cost != nullptr && index < sites.size()) [[unlikely]] { + cost->BeginCommand(sites[index]); + } command->Execute(cmdbuf, upload_cmdbuf); command->~Command(); command = next; } + sites.clear(); submit = false; command_offset = 0; first = nullptr; @@ -111,7 +123,9 @@ void Scheduler::CommandChunk::ExecuteAll(vk::CommandBuffer cmdbuf, Scheduler::Scheduler(const Device& device_, StateTracker& state_tracker_) : device{device_}, state_tracker{state_tracker_}, master_semaphore{std::make_unique(device)}, - command_pool{std::make_unique(*master_semaphore, device)} { + command_pool{std::make_unique(*master_semaphore, device)}, + upload_batch{std::make_unique()}, + reordered_uploads{std::make_unique()} { if (Common::DkbFrameTrace::Enabled()) { const auto physical = device.GetPhysical(); @@ -146,6 +160,12 @@ Scheduler::Scheduler(const Device& device_, StateTracker& state_tracker_) trace_timestamp_bits, trace_timestamp_period); } } + if (VideoCommon::ReadTrace::Enabled()) { + CreateReadTrace(); + } + gpu_cost = GpuCost::Create(device, trace_timestamp_bits, trace_timestamp_period, + static_cast(trace_query_pool)); + record_sites = gpu_cost != nullptr; AcquireNewChunk(); AllocateWorkerCommandBuffer(); @@ -153,26 +173,247 @@ Scheduler::Scheduler(const Device& device_, StateTracker& state_tracker_) } Scheduler::~Scheduler() { - if (trace_query_pool) { + if (trace_query_pool || read_trace) { WaitWorker(); worker_thread.request_stop(); worker_thread.join(); if (trace_gpu_tick != 0) { master_semaphore->Wait(trace_gpu_tick); } + // The read trace's query pools are in every submission. + if (read_trace && CurrentTick() > 1) { + master_semaphore->Wait(CurrentTick() - 1); + } + } +} + +/// TEST (EDEN_NX2_READ_TRACE): device timestamps that place guest reads on the GPU's timeline. +struct Scheduler::ReadTraceState { + /// Submissions timed at their start and end (two queries a slot), by tick. + static constexpr u32 TickSlots = 4096; + /// TraceMark timestamps, by handle. + static constexpr u32 MarkSlots = 4096; + + vk::QueryPool ticks; + vk::QueryPool marks; + /// vkGetCalibratedTimestamps{KHR,EXT}; null without them. + PFN_vkGetCalibratedTimestampsKHR calibrate{}; + double ns_per_tick{}; + u64 next_tick{1}; ///< Worker: the submission its command buffers go in + u64 next_mark{1}; ///< Recording thread + std::atomic issued_marks{}; + std::atomic submitted{}; ///< The last tick vkQueueSubmit returned for + /// When vkQueueSubmit returned (Now()) for the tick in submit_ticks + std::array, TickSlots> submit_ticks{}; + std::array, TickSlots> submit_times{}; +}; + +void Scheduler::CreateReadTrace() { + const auto physical = device.GetPhysical(); + const u32 valid_bits = + physical.GetQueueFamilyProperties()[device.GetGraphicsFamily()].timestampValidBits; + const float period = physical.GetProperties().limits.timestampPeriod; + if (valid_bits == 0 || period <= 0.0f) { + LOG_WARNING(Render_Vulkan, "Read trace: the graphics queue has no timestamps"); + return; + } + auto created = std::make_unique(); + created->ns_per_tick = static_cast(period); + const auto create_pool = [this](u32 count) { + return device.GetLogical().CreateQueryPool({ + .sType = VK_STRUCTURE_TYPE_QUERY_POOL_CREATE_INFO, + .pNext = nullptr, + .flags = 0, + .queryType = VK_QUERY_TYPE_TIMESTAMP, + .queryCount = count, + .pipelineStatistics = 0, + }); + }; + try { + created->ticks = create_pool(ReadTraceState::TickSlots * 2); + created->marks = create_pool(ReadTraceState::MarkSlots); + } catch (const vk::Exception& error) { + LOG_WARNING(Render_Vulkan, "Read trace: no query pools: {}", error.what()); + return; + } + // The device enables either extension with EDEN_NX2_READ_TRACE (vulkan_device.cpp). + const auto& dld = device.GetDispatchLoader(); + const auto& extensions = device.GetAvailableExtensions(); + const bool khr = extensions.contains(VK_KHR_CALIBRATED_TIMESTAMPS_EXTENSION_NAME); + if (khr || extensions.contains(VK_EXT_CALIBRATED_TIMESTAMPS_EXTENSION_NAME)) { + const auto get_domains = + reinterpret_cast( + dld.vkGetInstanceProcAddr(device.GetInstance(), + khr ? "vkGetPhysicalDeviceCalibrateableTimeDomainsKHR" + : "vkGetPhysicalDeviceCalibrateableTimeDomainsEXT")); + const auto calibrate = reinterpret_cast( + dld.vkGetDeviceProcAddr(*device.GetLogical(), khr ? "vkGetCalibratedTimestampsKHR" + : "vkGetCalibratedTimestampsEXT")); + std::array domains{}; + u32 count = static_cast(domains.size()); + if (get_domains != nullptr && calibrate != nullptr && + get_domains(physical, &count, domains.data()) >= VK_SUCCESS && + std::ranges::find(domains.begin(), domains.begin() + (std::min)(count, 8u), + VK_TIME_DOMAIN_DEVICE_KHR) != + domains.begin() + (std::min)(count, 8u)) { + created->calibrate = calibrate; + } + } + LOG_INFO(Render_Vulkan, "Read trace: device timestamps every {} ns, {} bits, {}", period, + valid_bits, + created->calibrate ? "device clock readable (calibrated timestamps)" + : "no calibrated timestamps: reads stay unplaced"); + read_trace = std::move(created); +} + +u64 Scheduler::TraceMark() { + if (!read_trace) { + return 0; + } + const u64 handle = read_trace->next_mark++; + read_trace->issued_marks.store(handle, std::memory_order_release); + RecordWithUploadBuffer( + [pool = *read_trace->marks, slot = static_cast(handle % ReadTraceState::MarkSlots)]( + vk::CommandBuffer cmdbuf, vk::CommandBuffer upload_cmdbuf) { + // The upload buffer runs first in the same submission. + upload_cmdbuf.ResetQueryPool(pool, slot, 1); + cmdbuf.WriteTimestamp(VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, pool, slot); + }); + return handle; +} + +void Scheduler::TraceRead(VideoCommon::ReadTrace::Read& read, VideoCommon::ReadTrace::Step step) { + namespace ReadTrace = VideoCommon::ReadTrace; + if (!read_trace) { + return; + } + switch (step) { + case ReadTrace::Step::Request: + master_semaphore->Refresh(); + read.completed = master_semaphore->KnownGpuTick(); + read.submitted = read_trace->submitted.load(std::memory_order_acquire); + read.recording = master_semaphore->CurrentTick(); + return; + case ReadTrace::Step::Calibrate: + if (read_trace->calibrate != nullptr) { + const VkCalibratedTimestampInfoKHR info{ + .sType = VK_STRUCTURE_TYPE_CALIBRATED_TIMESTAMP_INFO_KHR, + .pNext = nullptr, + .timeDomain = VK_TIME_DOMAIN_DEVICE_KHR, + }; + u64 value{}; + u64 deviation{}; + const u64 before = ReadTrace::Now(); + if (read_trace->calibrate(*device.GetLogical(), 1, &info, &value, &deviation) == + VK_SUCCESS) { + const u64 after = ReadTrace::Now(); + read.device = value; + read.host = before + (after - before) / 2; + read.uncertainty = after - before; + } + } + return; + case ReadTrace::Step::Done: + ReadTrace::Record(read, PlaceRead(read)); + return; + } +} + +VideoCommon::ReadTrace::Timeline Scheduler::PlaceRead( + const VideoCommon::ReadTrace::Read& read) const { + namespace ReadTrace = VideoCommon::ReadTrace; + if (!read.has_copy) { + return {}; + } + const ReadTraceState& trace = *read_trace; + const u64 copy_tick = read.copy.tick; + struct Result { + u64 value; + u64 available; + }; + const auto& logical = device.GetLogical(); + const auto mark = [&](u64 handle) -> std::optional { + const auto reused = [&] { + return trace.issued_marks.load(std::memory_order_acquire) - handle >= + ReadTraceState::MarkSlots; + }; + Result result{}; + if (handle == 0 || reused() || + logical.GetQueryResults( + *trace.marks, static_cast(handle % ReadTraceState::MarkSlots), 1, + sizeof(result), &result, sizeof(result), + VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WITH_AVAILABILITY_BIT) != VK_SUCCESS || + result.available == 0 || reused()) { + return std::nullopt; + } + return result.value; + }; + // The submissions after the last one done at the request, up to the copy's: all done now. + const auto place = [&]() -> ReadTrace::Timeline { + const u64 first_tick = read.completed + 1; + if (read.device == 0 || !read.has_write || copy_tick < first_tick || + copy_tick - first_tick >= ReadTraceState::TickSlots / 2) { + return {}; + } + const auto write_end = mark(read.write.mark); + const auto copy_start = mark(read.copy.start); + const auto copy_end = mark(read.copy.end); + if (!write_end || !copy_start || !copy_end) { + return {}; + } + const u32 tick_count = static_cast(copy_tick - first_tick + 1); + std::vector results(size_t{tick_count} * 2); + for (u32 done = 0; done < tick_count;) { + const u32 slot = static_cast((first_tick + done) % ReadTraceState::TickSlots); + const u32 count = (std::min)(tick_count - done, ReadTraceState::TickSlots - slot); + if (logical.GetQueryResults( + *trace.ticks, slot * 2, count * 2, size_t{count} * 2 * sizeof(Result), + &results[size_t{done} * 2], sizeof(Result), + VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WITH_AVAILABILITY_BIT) != VK_SUCCESS) { + return {}; + } + done += count; + } + std::vector> submissions; + submissions.reserve(tick_count); + for (u32 i = 0; i < tick_count; ++i) { + const Result& start = results[size_t{i} * 2]; + const Result& end = results[size_t{i} * 2 + 1]; + if (start.available == 0 || end.available == 0) { + return {}; + } + submissions.emplace_back(start.value, end.value); + } + return ReadTrace::Place(read, ReadTrace::DeviceTimes{ + .ns_per_tick = trace.ns_per_tick, + .write_end = *write_end, + .copy_start = *copy_start, + .copy_end = *copy_end, + .submissions = submissions, + }); + }; + ReadTrace::Timeline timeline = place(); + const u32 copy_slot = static_cast(copy_tick % ReadTraceState::TickSlots); + if (trace.submit_ticks[copy_slot].load(std::memory_order_acquire) == copy_tick) { + const u64 submitted_at = trace.submit_times[copy_slot].load(std::memory_order_relaxed); + if (submitted_at >= read.copy.flushed) { + timeline.has_submit = true; + timeline.submit = submitted_at - read.copy.flushed; + } } + return timeline; } u64 Scheduler::Flush(VkSemaphore signal_semaphore, VkSemaphore wait_semaphore, std::source_location location) { if (state.renderpass) { - CountRenderPassEnd(location.file_name(), location.line()); + CountRenderPassEnd(location); } if (VideoCommon::GpuSplit::check_owner) [[unlikely]] { VideoCommon::GpuSplit::CheckOwner("Scheduler::Flush"); } // When flushing, we only send data to the worker thread; no waiting is necessary. - const u64 signal_value = SubmitExecution(signal_semaphore, wait_semaphore); + const u64 signal_value = SubmitExecution(signal_semaphore, wait_semaphore, location); AllocateNewContext(); last_flush = std::chrono::steady_clock::now(); flush_pending = false; @@ -239,7 +480,7 @@ bool Scheduler::WaitSubmitted(u64 tick) { void Scheduler::Finish(VkSemaphore signal_semaphore, VkSemaphore wait_semaphore, std::source_location location) { if (state.renderpass) { - CountRenderPassEnd(location.file_name(), location.line()); + CountRenderPassEnd(location); } if (VideoCommon::GpuSplit::check_owner) [[unlikely]] { VideoCommon::GpuSplit::CheckOwner("Scheduler::Finish"); @@ -249,9 +490,11 @@ void Scheduler::Finish(VkSemaphore signal_semaphore, VkSemaphore wait_semaphore, const u64 presubmit_tick = CurrentTick(); const bool profile = VideoCommon::Nx2Profile::Enabled(); const u64 start = profile ? VideoCommon::Nx2Profile::Now() : 0; - SubmitExecution(signal_semaphore, wait_semaphore); + const u64 wait_start = VideoCommon::GpuWaitStats::Now(); + SubmitExecution(signal_semaphore, wait_semaphore, location); Wait(presubmit_tick); AllocateNewContext(); + VideoCommon::GpuWaitStats::AddHostWait(VideoCommon::GpuWaitStats::Now() - wait_start); last_flush = std::chrono::steady_clock::now(); flush_pending = false; if (profile) { @@ -284,6 +527,10 @@ void Scheduler::DispatchWork() { if (VideoCommon::GpuSplit::check_owner) [[unlikely]] { VideoCommon::GpuSplit::CheckOwner("Scheduler::DispatchWork"); } + if (uploads_pending) { + // What the worker gets is everything recorded so far. + FlushUploads(); + } if (chunk && !chunk->Empty()) { // Waking the worker is a system call (~2% of the GPU thread at one dispatch per 8 // draws): skip it while the worker is still busy, it pops the queue before waiting. @@ -300,11 +547,75 @@ void Scheduler::DispatchWork() { } } +void Scheduler::RecordUpload(VkBuffer src, VkBuffer dst, std::span copies, + std::source_location location) { + if (VideoCommon::GpuSplit::check_owner) [[unlikely]] { + VideoCommon::GpuSplit::CheckOwner("Scheduler::RecordUpload"); + } + if (copies.empty()) { + return; + } + if (uploads_pending && !upload_batch->CanAdd(src, dst, copies)) { + FlushUploads(); + } + if (!uploads_pending) { + upload_batch_site = location; + } + upload_batch->Add(src, dst, copies); + uploads_pending = true; +} + +void Scheduler::FlushUploads() { + // Copies, so the batch's vectors keep their capacity for the next one. + auto command = [copies = upload_batch->copies, + groups = upload_batch->groups](vk::CommandBuffer cmdbuf, vk::CommandBuffer) { + UploadBatch::Record(cmdbuf, copies, groups); + }; + upload_batch->copies.clear(); + upload_batch->groups.clear(); + uploads_pending = false; + RecordAs(std::move(command), upload_batch_site, RecordSite::Own, 0); +} + +void Scheduler::RecordReorderedUpload(VkBuffer src, VkBuffer dst, + std::span copies, u64 prior_tick) { + if (VideoCommon::GpuSplit::check_owner) [[unlikely]] { + VideoCommon::GpuSplit::CheckOwner("Scheduler::RecordReorderedUpload"); + } + if (copies.empty()) { + return; + } + reordered_uploads->Add(src, dst, copies, prior_tick); + reordered_uploads_pending = true; +} + +void Scheduler::FlushReorderedUploads() { + std::vector batch_copies; + std::vector batch_groups; + u64 prior_tick = 0; + reordered_uploads->Take(batch_copies, batch_groups, prior_tick); + reordered_uploads_pending = false; + // The copies must come after earlier submissions' work on their bytes: finished already, or + // waited for by a barrier. The host GPU is asked once a submission, when it may still run it. + bool batch_after_prior = prior_tick != 0 && !master_semaphore->IsFree(prior_tick); + if (batch_after_prior) { + master_semaphore->Refresh(); + batch_after_prior = !master_semaphore->IsFree(prior_tick); + } + RecordAs( + [copies = std::move(batch_copies), groups = std::move(batch_groups), + after_prior_work = batch_after_prior](vk::CommandBuffer, vk::CommandBuffer upload_cmdbuf) { + ReorderedUploadBatch::Record(upload_cmdbuf, copies, groups, after_prior_work); + }, + std::source_location::current(), RecordSite::Own, 0); +} + void Scheduler::BeginRenderPassImpl(const Framebuffer* framebuffer, VkRenderPass renderpass, const VkClearValue* clear_values, u32 clear_value_count) { if (VideoCommon::Nx2Profile::Enabled()) { g_profile_renderpasses.fetch_add(1, std::memory_order_relaxed); } + ++trace_counts.render_passes; const VkFramebuffer framebuffer_handle = framebuffer->Handle(); const VkExtent2D render_area = framebuffer->RenderArea(); ++render_pass_serial; @@ -324,8 +635,8 @@ void Scheduler::BeginRenderPassImpl(const Framebuffer* framebuffer, VkRenderPass values[i] = clear_values[i]; } const bool after_compute = std::exchange(compute_writes_pending, false); - Record([renderpass, framebuffer_handle, render_area, values, clear_value_count, - after_compute](vk::CommandBuffer cmdbuf) { + RecordAs([renderpass, framebuffer_handle, render_area, values, clear_value_count, + after_compute](vk::CommandBuffer cmdbuf, vk::CommandBuffer) { const VkRenderPassBeginInfo renderpass_bi{ .sType = VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO, .pNext = nullptr, @@ -352,7 +663,7 @@ void Scheduler::BeginRenderPassImpl(const Framebuffer* framebuffer, VkRenderPass VK_PIPELINE_STAGE_ALL_GRAPHICS_BIT, 0, COMPUTE_TO_GRAPHICS); } cmdbuf.BeginRenderPass(renderpass_bi, VK_SUBPASS_CONTENTS_INLINE); - }); + }, last_pass_end, RecordSite::PassBegin, 0); num_renderpass_images = framebuffer->NumImages(); renderpass_images = framebuffer->Images(); renderpass_image_ranges = framebuffer->ImageRanges(); @@ -446,18 +757,18 @@ void Scheduler::RequestRenderpass(const Framebuffer* framebuffer, std::source_lo return; } if (state.renderpass) { - CountRenderPassEnd(location.file_name(), location.line()); + CountRenderPassEnd(location); } // Ends any active pass and realizes a deferred clear - EndRenderPass(); + EndRenderPass(location); BeginRenderPassImpl(framebuffer, renderpass, nullptr, 0); } void Scheduler::RequestOutsideRenderPassOperationContext(std::source_location location) { if (state.renderpass) { - CountRenderPassEnd(location.file_name(), location.line()); + CountRenderPassEnd(location); } - EndRenderPass(); + EndRenderPass(location); } bool Scheduler::UpdateGraphicsPipeline(GraphicsPipeline* pipeline) { @@ -554,7 +865,8 @@ void Scheduler::WorkerThread(std::stop_token stop_token) { // Perform the work, tracking whether the chunk was a submission // before executing. const bool has_submit = work->HasSubmit(); - work->ExecuteAll(current_cmdbuf, current_upload_cmdbuf); + work->ExecuteAll(current_cmdbuf, current_upload_cmdbuf, + trace_gpu_recording ? gpu_cost.get() : nullptr); // If the chunk was a submission, reallocate the command buffer. if (has_submit) { @@ -587,8 +899,18 @@ void Scheduler::AllocateWorkerCommandBuffer() { .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT, .pInheritanceInfo = nullptr, }); + if (read_trace) { + // When the host GPU starts the submission these go in (its end: SubmitExecution). + const u32 query = static_cast(read_trace->next_tick % ReadTraceState::TickSlots) * 2; + current_upload_cmdbuf.ResetQueryPool(*read_trace->ticks, query, 2); + current_upload_cmdbuf.WriteTimestamp(VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, *read_trace->ticks, + query); + } if (trace_query_pool && !trace_gpu_failed) { TraceGpuSubmission(); + if (gpu_cost && trace_gpu_recording) { + gpu_cost->BeginSubmission(current_cmdbuf, current_upload_cmdbuf, trace_gpu_count == 0); + } } } @@ -653,6 +975,17 @@ void Scheduler::ReportWorkTimes() { if (result != VK_SUCCESS) { return; } + if (read_trace && count < TraceMaxWork) { + // Every draw and dispatch of the frame was timed: the reads placed in it split by them. + std::vector> work; + work.reserve(count); + for (u32 i = 0; i < count; ++i) { + if (results[i * 2].available != 0 && results[i * 2 + 1].available != 0) { + work.emplace_back(results[i * 2].timestamp, results[i * 2 + 1].timestamp); + } + } + VideoCommon::ReadTrace::NoteTimedWork(work, read_trace->ns_per_tick); + } struct Totals { double total_ms{}; double max_ms{}; @@ -770,6 +1103,9 @@ void Scheduler::TraceGpuSubmission() { work.dispatch_attempts, shaders, work.untracked_uses); } ReportWorkTimes(); + if (gpu_cost) { + gpu_cost->FrameDone(); + } trace_gpu_tick = 0; trace_gpu_count = 0; trace_gpu_skipped = 0; @@ -789,17 +1125,26 @@ void Scheduler::TraceGpuSubmission() { trace_gpu_recording = true; } -u64 Scheduler::SubmitExecution(VkSemaphore signal_semaphore, VkSemaphore wait_semaphore) { +u64 Scheduler::SubmitExecution(VkSemaphore signal_semaphore, VkSemaphore wait_semaphore, + std::source_location cause) { + if (uploads_pending) { + // In the tick they were recorded at. + FlushUploads(); + } if (VideoCommon::Nx2Profile::Enabled()) { g_profile_submits.fetch_add(1, std::memory_order_relaxed); ReportSubmitCounts(); } - EndPendingOperations(); + EndPendingOperations(cause); InvalidateState(); + if (reordered_uploads_pending) { + // Into the upload command buffer of the submission their uploads were decided for. + FlushReorderedUploads(); + } const u64 signal_value = master_semaphore->NextTick(); - RecordWithUploadBuffer([signal_semaphore, wait_semaphore, signal_value, - this](vk::CommandBuffer cmdbuf, vk::CommandBuffer upload_cmdbuf) { + RecordAs([signal_semaphore, wait_semaphore, signal_value, + this](vk::CommandBuffer cmdbuf, vk::CommandBuffer upload_cmdbuf) { static constexpr VkMemoryBarrier WRITE_BARRIER{ .sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER, .pNext = nullptr, @@ -809,6 +1154,9 @@ u64 Scheduler::SubmitExecution(VkSemaphore signal_semaphore, VkSemaphore wait_se upload_cmdbuf.PipelineBarrier(VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, 0, WRITE_BARRIER); upload_cmdbuf.End(); if (trace_gpu_recording) { + if (gpu_cost) { + gpu_cost->EndSubmission(cmdbuf); + } cmdbuf.WriteTimestamp(VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, *trace_query_pool, trace_gpu_count * 2 + 1); trace_gpu_tick = signal_value; @@ -822,6 +1170,10 @@ u64 Scheduler::SubmitExecution(VkSemaphore signal_semaphore, VkSemaphore wait_se if (trace_query_pool) { trace_work = {}; } + if (read_trace) { + const u32 query = static_cast(signal_value % ReadTraceState::TickSlots) * 2 + 1; + cmdbuf.WriteTimestamp(VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, *read_trace->ticks, query); + } cmdbuf.End(); if (on_submit) { @@ -846,6 +1198,14 @@ u64 Scheduler::SubmitExecution(VkSemaphore signal_semaphore, VkSemaphore wait_se vk::Check(result); break; } + if (read_trace) { + const u32 slot = static_cast(signal_value % ReadTraceState::TickSlots); + read_trace->submit_times[slot].store(VideoCommon::ReadTrace::Now(), + std::memory_order_relaxed); + read_trace->submit_ticks[slot].store(signal_value, std::memory_order_release); + read_trace->submitted.store(signal_value, std::memory_order_release); + read_trace->next_tick = signal_value + 1; + } if (trace_frame_end != 0) { if (trace_gpu_sampling) { trace_gpu_pending = true; @@ -858,7 +1218,7 @@ u64 Scheduler::SubmitExecution(VkSemaphore signal_semaphore, VkSemaphore wait_se } trace_frame_end = 0; } - }); + }, cause, RecordSite::Submit, 0); chunk->MarkSubmit(); DispatchWork(); return signal_value; @@ -876,14 +1236,14 @@ void Scheduler::InvalidateState() { state_tracker.InvalidateCommandBufferState(); } -void Scheduler::EndPendingOperations() { +void Scheduler::EndPendingOperations(std::source_location cause) { // Pause, don't reset: a submission can land between a guest's counter reset and its report, // and resetting here would drop the samples counted so far (the report then reads too low). query_cache->CounterClose(VideoCommon::QueryType::ZPassPixelCount64); - EndRenderPass(); + EndRenderPass(cause); } -void Scheduler::EndRenderPass() +void Scheduler::EndRenderPass(std::source_location cause) { RealizeDeferredClear(); if (!state.renderpass) { @@ -891,7 +1251,10 @@ void Scheduler::EndRenderPass() } ++recording_epoch; fragment_barrier.Reset(); + const u8 site_flags = + static_cast(render_pass_has_stores ? RecordSite::PassHadStores : 0); render_pass_has_stores = false; + last_pass_end = cause; ++render_pass_serial; query_cache->CounterClose(VideoCommon::QueryType::StreamingByteCount); @@ -905,11 +1268,11 @@ void Scheduler::EndRenderPass() query_cache->CounterEnable(VideoCommon::QueryType::ZPassPixelCount64, false); query_cache->NotifySegment(false); - Record([num_images = num_renderpass_images, + RecordAs([num_images = num_renderpass_images, images = renderpass_images, ranges = renderpass_image_ranges, has_transform_feedback = device.IsExtTransformFeedbackSupported()]( - vk::CommandBuffer cmdbuf) { + vk::CommandBuffer cmdbuf, vk::CommandBuffer) { std::array barriers; for (size_t i = 0; i < num_images; ++i) { const VkImageSubresourceRange& range = ranges[i]; @@ -960,7 +1323,7 @@ void Scheduler::EndRenderPass() VK_PIPELINE_STAGE_VERTEX_INPUT_BIT | VK_PIPELINE_STAGE_TRANSFER_BIT, 0, XFB_OUTPUT_BARRIER); } - }); + }, cause, RecordSite::PassEnd, site_flags); state.renderpass = VkRenderPass{}; num_renderpass_images = 0; diff --git a/src/video_core/renderer_vulkan/vk_scheduler.h b/src/video_core/renderer_vulkan/vk_scheduler.h index b3f5708312..5b1826bfd4 100644 --- a/src/video_core/renderer_vulkan/vk_scheduler.h +++ b/src/video_core/renderer_vulkan/vk_scheduler.h @@ -25,6 +25,7 @@ #include "common/polyfill_thread.h" #include "common/settings.h" #include "video_core/gpu_split.h" +#include "video_core/nx2_read_trace.h" #include "video_core/renderer_vulkan/fragment_barrier_state.h" #include "video_core/renderer_vulkan/vk_master_semaphore.h" #include "video_core/vulkan_common/vulkan_wrapper.h" @@ -39,11 +40,33 @@ namespace Vulkan { class CommandPool; class Device; class Framebuffer; +class GpuCost; class GraphicsPipeline; +class ReorderedUploadBatch; class StateTracker; +class UploadBatch; struct QueryCacheParams; +/// TEST (EDEN_NX2_GPU_COST, vk_gpu_cost.h): the source line that recorded a command. A render +/// pass's end carries the line that ended it, its begin the line that ended the pass before. +struct RecordSite { + enum Role : u8 { + Own, + PassEnd, + PassBegin, + Submit, + }; + enum Flags : u8 { + PassHadStores = 1 << 0, ///< PassEnd: a draw in the pass wrote storage buffers or images + }; + const char* file; + const char* function; + u32 line; + Role role; + u8 flags; +}; + /// The scheduler abstracts command buffer and fence management with an interface that's able to do /// OpenGL-like operations on Vulkan command buffers. class Scheduler { @@ -98,6 +121,19 @@ class Scheduler { cmdbuf.WriteTimestamp(VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, *trace_work_pool, query + 1); } + /// TEST (EDEN_NX2_READ_TRACE, nx2_read_trace.h), recording thread: a device timestamp written + /// once the work recorded so far is done. Returns its handle for TraceRead, 0 when off. + u64 TraceMark(); + /// Recording thread: the draws, dispatches and render passes recorded so far. + [[nodiscard]] VideoCommon::ReadTrace::Counts TraceCounts() const noexcept { + return trace_counts; + } + void TraceCountWork(bool dispatch) noexcept { + ++(dispatch ? trace_counts.dispatches : trace_counts.draws); + } + /// A reading thread's step of a guest read that waits. + void TraceRead(VideoCommon::ReadTrace::Read& read, VideoCommon::ReadTrace::Step step); + /// Sends currently recorded work to the worker thread. void DispatchWork(); @@ -186,27 +222,38 @@ class Scheduler { on_submit = std::move(func); } + /// An upload in order: copies into dst after the work recorded before them and before the + /// work recorded after them (the barriers of an in-order buffer copy). Uploads with no other + /// command between them share one command and one pair of barriers, unless one would write + /// bytes another writes, or a buffer another reads. Call outside a render pass. + void RecordUpload(VkBuffer src, VkBuffer dst, std::span copies, + std::source_location location = std::source_location::current()); + + /// An upload ahead of all the work of this submission's main command buffer: it runs in the + /// upload command buffer, which the submission's reordered uploads are recorded into as one + /// command when it is submitted. The caller guarantees that no work recorded in this + /// submission uses the bytes it writes, nor work of an earlier submission after prior_tick + /// (0: of any). Unless the host GPU has finished submission prior_tick by then, the upload + /// command buffer begins with a barrier on the work of earlier submissions. + void RecordReorderedUpload(VkBuffer src, VkBuffer dst, std::span copies, + u64 prior_tick); + /// Send work to a separate thread. template requires std::is_invocable_v - void RecordWithUploadBuffer(T&& command) { - if (VideoCommon::GpuSplit::check_owner) [[unlikely]] { - VideoCommon::GpuSplit::CheckOwner("Scheduler::Record"); - } - if (chunk->Record(command)) { - return; - } - DispatchWork(); - (void)chunk->Record(command); + void RecordWithUploadBuffer(T&& command, + std::source_location location = std::source_location::current()) { + RecordAs(std::forward(command), location, RecordSite::Own, 0); } template requires std::is_invocable_v - void Record(T&& c) { - this->RecordWithUploadBuffer( + void Record(T&& c, std::source_location location = std::source_location::current()) { + RecordAs( [command = std::move(c)](vk::CommandBuffer cmdbuf, vk::CommandBuffer) { command(cmdbuf); - }); + }, + location, RecordSite::Own, 0); } /// Whether the recording thread submitted `tick` within 20 ms. @@ -362,12 +409,20 @@ class Scheduler { class CommandChunk final { public: - void ExecuteAll(vk::CommandBuffer cmdbuf, vk::CommandBuffer upload_cmdbuf); + /// cost: the TEST GPU cost trace when this chunk is in a submission it samples. + void ExecuteAll(vk::CommandBuffer cmdbuf, vk::CommandBuffer upload_cmdbuf, GpuCost* cost); + + /// TEST (EDEN_NX2_GPU_COST): the site of each command, in order. + void NoteSite(const RecordSite& site) { + sites.push_back(site); + } template bool Record(T& command) { using FuncType = TypedCommand; static_assert(sizeof(FuncType) < sizeof(data), "Lambda is too large"); + // The offset is aligned within data, so data itself must be aligned at least as much. + static_assert(alignof(FuncType) <= DataAlignment, "Lambda needs more alignment"); command_offset = Common::AlignUp(command_offset, alignof(FuncType)); if (command_offset > sizeof(data) - sizeof(FuncType)) { @@ -403,9 +458,35 @@ class Scheduler { size_t command_offset = 0; bool submit = false; - alignas(std::max_align_t) std::array data{}; + std::vector sites; + /// Commands are constructed in data at offsets aligned for their type. Some hold + /// alignas(16) members (the ASTC decoder pass's BlockLinearSwizzle2DParams), and + /// std::max_align_t is only 8 on MSVC and clang-cl, so data asks for 16 itself rather + /// than depending on where the members before it happen to end. + static constexpr size_t DataAlignment = + alignof(std::max_align_t) > 16 ? alignof(std::max_align_t) : 16; + alignas(DataAlignment) std::array data{}; }; + template + void RecordAs(T&& command, std::source_location location, RecordSite::Role role, u8 flags) { + if (VideoCommon::GpuSplit::check_owner) [[unlikely]] { + VideoCommon::GpuSplit::CheckOwner("Scheduler::Record"); + } + if (uploads_pending) { + // Uploads recorded before this command go before it. + FlushUploads(); + } + if (!chunk->Record(command)) { + DispatchWork(); + (void)chunk->Record(command); + } + if (record_sites) [[unlikely]] { + chunk->NoteSite({location.file_name(), location.function_name(), location.line(), role, + flags}); + } + } + struct State { VkRenderPass renderpass{}; VkFramebuffer framebuffer{}; @@ -442,13 +523,21 @@ class Scheduler { void ReportWorkTimes(); void TraceShader(u64 hash, u32 stage); - u64 SubmitExecution(VkSemaphore signal_semaphore, VkSemaphore wait_semaphore); + struct ReadTraceState; + void CreateReadTrace(); + [[nodiscard]] VideoCommon::ReadTrace::Timeline PlaceRead( + const VideoCommon::ReadTrace::Read& read) const; + + /// cause: the line that asked for the submission, which ends a render pass still open. + u64 SubmitExecution(VkSemaphore signal_semaphore, VkSemaphore wait_semaphore, + std::source_location cause); void AllocateNewContext(); - void EndPendingOperations(); + void EndPendingOperations(std::source_location cause); - void EndRenderPass(); + /// cause: the line that needs the render pass ended (TEST EDEN_NX2_GPU_COST's site of it). + void EndRenderPass(std::source_location cause = std::source_location::current()); void AcquireNewChunk(); @@ -502,6 +591,28 @@ class Scheduler { bool trace_gpu_pending{}; bool trace_gpu_failed{}; + /// TEST: EDEN_NX2_READ_TRACE + std::unique_ptr read_trace; + VideoCommon::ReadTrace::Counts trace_counts{}; + + /// TEST: EDEN_NX2_GPU_COST. record_sites: every command notes its RecordSite. + std::unique_ptr gpu_cost; + bool record_sites{}; + + /// RecordUpload's copies not recorded yet; uploads_pending while it holds any. + std::unique_ptr upload_batch; + std::source_location upload_batch_site{}; + bool uploads_pending{}; + /// Records the pending uploads as one command. + void FlushUploads(); + /// RecordReorderedUpload's copies, recorded when the submission is. + std::unique_ptr reordered_uploads; + bool reordered_uploads_pending{}; + /// Records the reordered uploads into the upload command buffer as one command. + void FlushReorderedUploads(); + /// The line that ended the last render pass: the site of the next pass's begin. + std::source_location last_pass_end{}; + VideoCommon::QueryCacheBase* query_cache = nullptr; vk::CommandBuffer current_cmdbuf; diff --git a/src/video_core/renderer_vulkan/vk_staging_buffer_pool.cpp b/src/video_core/renderer_vulkan/vk_staging_buffer_pool.cpp index 1000aed081..34047305f0 100644 --- a/src/video_core/renderer_vulkan/vk_staging_buffer_pool.cpp +++ b/src/video_core/renderer_vulkan/vk_staging_buffer_pool.cpp @@ -96,8 +96,11 @@ StagingBufferPool::StagingBufferPool(const Device& device_, MemoryAllocator& mem .pNext = nullptr, .flags = 0, .size = stream_buffer_size + STREAM_BUFFER_TAIL, - .usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | - VK_BUFFER_USAGE_INDEX_BUFFER_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT, + // Transfer destination too: GPU-written list words are copied over their placeholders + // in the constant buffers streamed here (Nx2ListSlots::PatchUniform). + .usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT | + VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_INDEX_BUFFER_BIT | + VK_BUFFER_USAGE_STORAGE_BUFFER_BIT, .sharingMode = VK_SHARING_MODE_EXCLUSIVE, .queueFamilyIndexCount = 0, .pQueueFamilyIndices = nullptr, diff --git a/src/video_core/renderer_vulkan/vk_texture_cache.cpp b/src/video_core/renderer_vulkan/vk_texture_cache.cpp index aa25dfe8f5..ba66de32ed 100644 --- a/src/video_core/renderer_vulkan/vk_texture_cache.cpp +++ b/src/video_core/renderer_vulkan/vk_texture_cache.cpp @@ -2841,6 +2841,14 @@ u64 TextureCacheRuntime::GetDeviceMemoryUsage() const { return device.GetDeviceMemoryUsage(); } +u64 TextureCacheRuntime::GetRecycledImageMemory() const { + return vk::ImageRecycler::KeptBytes(); +} + +void TextureCacheRuntime::ShrinkRecycledImages(u64 bytes) { + vk::ImageRecycler::Shrink(bytes); +} + u64 TextureCacheRuntime::CurrentTick() const noexcept { return scheduler.CurrentTick(); } diff --git a/src/video_core/renderer_vulkan/vk_texture_cache.h b/src/video_core/renderer_vulkan/vk_texture_cache.h index bf230a5941..3726d9af62 100644 --- a/src/video_core/renderer_vulkan/vk_texture_cache.h +++ b/src/video_core/renderer_vulkan/vk_texture_cache.h @@ -101,6 +101,11 @@ class TextureCacheRuntime { u64 GetDeviceMemoryUsage() const; + /// Device memory the image recycler keeps (vk::ImageRecycler): the collector counts it as + /// free, and when memory runs short the recycler gives it back first. + u64 GetRecycledImageMemory() const; + void ShrinkRecycledImages(u64 bytes); + /// GPU progress: the texture cache frees a deleted image once the GPU is past its last use. [[nodiscard]] u64 CurrentTick() const noexcept; diff --git a/src/video_core/renderer_vulkan/vk_upload_reorder_check.cpp b/src/video_core/renderer_vulkan/vk_upload_reorder_check.cpp new file mode 100644 index 0000000000..0d3b8731ec --- /dev/null +++ b/src/video_core/renderer_vulkan/vk_upload_reorder_check.cpp @@ -0,0 +1,292 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "common/logging.h" +#include "video_core/renderer_vulkan/vk_scheduler.h" +#include "video_core/renderer_vulkan/vk_upload_reorder_check.h" +#include "video_core/texture_cache/types.h" + +namespace Vulkan { +namespace { + +constexpr size_t NUM_SOURCES = static_cast(UploadReorderCheck::Source::Count); +constexpr std::array SOURCE_NAMES{ + "a marked use", "a descriptor", + "a texel buffer", "a vertex buffer", + "an index buffer", "transform feedback", + "a multi-range source", "a copy reading", + "a copy writing", "a clear", +}; +static_assert(SOURCE_NAMES[NUM_SOURCES - 1] != nullptr); + +struct Range { + u64 end; + u32 sources; ///< bit per Source +}; + +/// Byte ranges by their begin, disjoint. Overlapping ranges merge, their sources too; touching +/// ones merge when they have the same sources, so an error names what used the bytes. +using RangeSet = std::map; + +void Insert(RangeSet& set, u64 begin, u64 end, u32 sources) { + auto it = set.upper_bound(begin); + if (it != set.begin()) { + const auto before = std::prev(it); + if (before->second.end >= end && (before->second.sources & sources) == sources) { + // Noted already (draws bind the same ranges again and again). + return; + } + if (before->second.end > begin || + (before->second.end == begin && before->second.sources == sources)) { + it = before; + } + } + u64 merged_begin = begin; + u64 merged_end = end; + u32 merged_sources = sources; + while (it != set.end() && (it->first < merged_end || + (it->first == merged_end && it->second.sources == merged_sources))) { + merged_begin = (std::min)(merged_begin, it->first); + merged_end = (std::max)(merged_end, it->second.end); + merged_sources |= it->second.sources; + it = set.erase(it); + } + set.emplace(merged_begin, Range{merged_end, merged_sources}); +} + +/// The sources of the ranges that share a byte with [begin, end), 0 when none does. +u32 Overlap(const RangeSet& set, u64 begin, u64 end) { + auto it = set.upper_bound(begin); + if (it != set.begin()) { + --it; + } + u32 sources = 0; + for (; it != set.end() && it->first < end; ++it) { + if (it->second.end > begin) { + sources |= it->second.sources; + } + } + return sources; +} + +/// A handle's value for the log: a pointer on 64-bit hosts, an integer on 32-bit ones. +template +u64 HandleValue(Handle handle) { + if constexpr (std::is_pointer_v) { + return static_cast(reinterpret_cast(handle)); + } else { + return static_cast(handle); + } +} + +std::string SourceList(u32 sources) { + std::string list; + for (size_t i = 0; i < NUM_SOURCES; ++i) { + if ((sources >> i) & 1) { + list += fmt::format("{}{}", list.empty() ? "" : ", ", SOURCE_NAMES[i]); + } + } + return list; +} + +struct Counts { + u64 uploads; + u64 reordered; + u64 reordered_after_unfinished; + u64 in_order; + u64 in_order_used; + u64 past_use; + u64 unordered; +}; + +} // Anonymous namespace + +struct UploadReorderCheck::Impl { + explicit Impl(Scheduler& scheduler_) : scheduler{scheduler_} {} + + struct TickUses { + u64 tick; + RangeSet ranges; + }; + struct BufferUses { + /// Oldest first, one entry a tick. + std::vector ticks; + }; + + /// Drops the uses of submissions the host GPU finished: nothing recorded now races them. + void Prune(BufferUses& uses) const { + const u64 current = scheduler.CurrentTick(); + auto it = uses.ticks.begin(); + while (it != uses.ticks.end() && it->tick < current && scheduler.IsFree(it->tick)) { + ++it; + } + uses.ticks.erase(uses.ticks.begin(), it); + } + + void Sweep() { + for (auto it = buffers.begin(); it != buffers.end();) { + Prune(it->second); + it = it->second.ticks.empty() ? buffers.erase(it) : std::next(it); + } + } + + void Report(std::chrono::steady_clock::time_point now) { + const double seconds = std::chrono::duration(now - window_start).count(); + LOG_INFO(Render_Vulkan, + "Upload reorder check in {:.1f} s: {} uploads from the stream buffer: {} " + "reordered ({} of them after unfinished work of earlier submissions on their " + "bytes), {} in order ({} after a use of their bytes in the same submission); " + "errors: {} reordered past a use of their bytes, {} reordered ahead of unfinished " + "earlier work on their bytes without waiting for it (since start: {} and {} in {} " + "uploads)", + seconds, window.uploads, window.reordered, window.reordered_after_unfinished, + window.in_order, window.in_order_used, window.past_use, window.unordered, + total.past_use, total.unordered, total.uploads); + window = {}; + window_start = now; + } + + Scheduler& scheduler; + std::mutex mutex; + std::unordered_map buffers; + u64 notes_since_sweep{}; + Counts window{}; + Counts total{}; + u64 errors_logged{}; + std::chrono::steady_clock::time_point window_start = std::chrono::steady_clock::now(); +}; + +std::unique_ptr UploadReorderCheck::Create(Scheduler& scheduler) { + const char* const value = std::getenv("EDEN_NX2_UPLOAD_REORDER_CHECK"); + if (value == nullptr || value[0] == '\0' || value[0] == '0') { + return nullptr; + } + std::unique_ptr check{new UploadReorderCheck(scheduler)}; + active = check.get(); + LOG_INFO(Render_Vulkan, + "Upload reorder check (EDEN_NX2_UPLOAD_REORDER_CHECK): checking the place of every " + "upload from the stream buffer against an exact model of the bytes each " + "submission's work uses"); + return check; +} + +UploadReorderCheck::UploadReorderCheck(Scheduler& scheduler) + : impl{std::make_unique(scheduler)} {} + +UploadReorderCheck::~UploadReorderCheck() { + if (active == this) { + active = nullptr; + } +} + +void UploadReorderCheck::NoteUse(VkBuffer buffer, u64 offset, u64 size, Source source) { + if (buffer == VK_NULL_HANDLE || size == 0) { + return; + } + Impl& self = *impl; + std::scoped_lock lock{self.mutex}; + const u64 tick = self.scheduler.CurrentTick(); + Impl::BufferUses& uses = self.buffers[buffer]; + self.Prune(uses); + if (uses.ticks.empty() || uses.ticks.back().tick < tick) { + uses.ticks.push_back(Impl::TickUses{tick, {}}); + } + // A whole-size binding ends at the end of the address space instead of wrapping around. + const u64 end = size > ~u64{0} - offset ? ~u64{0} : offset + size; + Insert(uses.ticks.back().ranges, offset, end, 1u << static_cast(source)); + if (++self.notes_since_sweep >= 1u << 16) { + self.notes_since_sweep = 0; + self.Sweep(); + } +} + +void UploadReorderCheck::CheckUpload(VkBuffer buffer, + std::span copies, + VideoCommon::UploadOrder order, std::source_location site) { + if (copies.empty()) { + return; + } + Impl& self = *impl; + std::scoped_lock lock{self.mutex}; + const u64 tick = self.scheduler.CurrentTick(); + // What work recorded before this upload in its submission uses of its bytes, and the newest + // earlier submission not known finished whose work does. + u32 used_now = 0; + u64 unfinished = 0; + const VideoCommon::BufferCopy* used_copy = nullptr; + if (const auto it = self.buffers.find(buffer); it != self.buffers.end()) { + self.Prune(it->second); + for (const Impl::TickUses& entry : it->second.ticks) { + for (const VideoCommon::BufferCopy& copy : copies) { + const u32 sources = + Overlap(entry.ranges, copy.dst_offset, copy.dst_offset + copy.size); + if (sources == 0) { + continue; + } + if (entry.tick == tick) { + used_now |= sources; + used_copy = used_copy ? used_copy : © + } else if (entry.tick < tick && !self.scheduler.IsFree(entry.tick)) { + unfinished = (std::max)(unfinished, entry.tick); + } + } + } + } + // A reordered upload waits for earlier work only through the submission's barrier, which is + // left out once the host GPU has finished order.prior_tick: no unfinished work after that + // submission may use its bytes. + const bool reordered = order.reorder; + const bool past_use = reordered && used_now != 0; + const bool unordered = reordered && unfinished > order.prior_tick; + for (Counts* const counts : {&self.window, &self.total}) { + ++counts->uploads; + if (reordered) { + ++counts->reordered; + counts->reordered_after_unfinished += unfinished != 0 ? 1 : 0; + } else { + ++counts->in_order; + counts->in_order_used += used_now != 0 ? 1 : 0; + } + counts->past_use += past_use ? 1 : 0; + counts->unordered += unordered ? 1 : 0; + } + if ((past_use || unordered) && + (self.errors_logged++ < 64 || (self.errors_logged & 1023) == 0)) { + const VideoCommon::BufferCopy& copy = used_copy ? *used_copy : copies.front(); + if (past_use) { + LOG_ERROR(Render_Vulkan, + "Upload reorder check: an upload into buffer {:#x} [{:#x}, {:#x}) at {}:{} " + "was put in the upload command buffer of submission {}, ahead of {} of its " + "bytes recorded before it in that submission", + HandleValue(buffer), copy.dst_offset, copy.dst_offset + copy.size, + site.file_name(), site.line(), tick, SourceList(used_now)); + } else { + LOG_ERROR(Render_Vulkan, + "Upload reorder check: an upload into buffer {:#x} at {}:{} was put in the " + "upload command buffer of submission {} after submission {} only, while " + "work of submission {} using its bytes is not known finished", + HandleValue(buffer), site.file_name(), site.line(), tick, order.prior_tick, + unfinished); + } + } + const auto now = std::chrono::steady_clock::now(); + if (now - self.window_start >= std::chrono::seconds{10}) { + self.Report(now); + } +} + +} // namespace Vulkan diff --git a/src/video_core/renderer_vulkan/vk_upload_reorder_check.h b/src/video_core/renderer_vulkan/vk_upload_reorder_check.h new file mode 100644 index 0000000000..f3e1079d3d --- /dev/null +++ b/src/video_core/renderer_vulkan/vk_upload_reorder_check.h @@ -0,0 +1,75 @@ +// SPDX-FileCopyrightText: Copyright 2026 Eden Emulator Project +// SPDX-License-Identifier: GPL-3.0-or-later + +#pragma once + +#include +#include +#include + +#include "common/common_types.h" +#include "video_core/buffer_cache/upload_order.h" +#include "video_core/vulkan_common/vulkan_wrapper.h" + +namespace VideoCommon { +struct BufferCopy; +} + +namespace Vulkan { + +class Scheduler; + +/// TEST (EDEN_NX2_UPLOAD_REORDER_CHECK=1): checks the place of every upload from the stream +/// buffer against an exact model of which bytes of which buffers the work recorded in each +/// submission uses. The model is kept apart from the buffers' own tracking (SubmissionUses): it +/// hears of every use the runtime records (descriptor, texel, vertex, index and transform +/// feedback bindings, multi-range sources, copies, clears) and of every use the buffer cache +/// marks (Buffer::MarkUsage), byte for byte. It counts and logs: +/// - an upload placed in the upload command buffer although work recorded before it in the same +/// submission uses a byte it writes (it would run before that work); +/// - one placed there without the barrier on earlier submissions' work although such work, not +/// known finished, uses a byte it writes. +/// Both must stay 0. +class UploadReorderCheck { +public: + enum class Source : u8 { + Marked, + Descriptor, + Texel, + Vertex, + Index, + TransformFeedback, + MultiRange, + CopyRead, + CopyWrite, + Clear, + Count, + }; + + /// Null when the variable is unset. + [[nodiscard]] static std::unique_ptr Create(Scheduler& scheduler); + + ~UploadReorderCheck(); + + /// The check in use, null when off. + [[nodiscard]] static UploadReorderCheck* Active() noexcept { + return active; + } + + /// Work recorded now uses [offset, offset + size) of buffer. + void NoteUse(VkBuffer buffer, u64 offset, u64 size, Source source); + + /// An upload from the stream buffer writing `copies` of buffer is recorded with `order`. + void CheckUpload(VkBuffer buffer, std::span copies, + VideoCommon::UploadOrder order, std::source_location site); + +private: + explicit UploadReorderCheck(Scheduler& scheduler); + + struct Impl; + std::unique_ptr impl; + + static inline UploadReorderCheck* active = nullptr; +}; + +} // namespace Vulkan diff --git a/src/video_core/shader_environment.cpp b/src/video_core/shader_environment.cpp index 40f8188aab..8c36e4be2f 100644 --- a/src/video_core/shader_environment.cpp +++ b/src/video_core/shader_environment.cpp @@ -491,11 +491,7 @@ Tegra::Texture::TICEntry GenericEnvironment::ReadTextureInfo(GPUVAddr tic_addr, return entry; } const GPUVAddr descriptor_addr{tic_addr + index * sizeof(Tegra::Texture::TICEntry)}; - if (texture_pool_flushed) { - gpu_memory->ReadBlockUnsafe(descriptor_addr, &entry, sizeof(entry)); - } else { - gpu_memory->ReadBlock(descriptor_addr, &entry, sizeof(entry)); - } + gpu_memory->ReadBlockUnsafe(descriptor_addr, &entry, sizeof(entry)); if (entry.format.Value() == Tegra::Texture::TextureFormat{} && std::getenv("EDEN_NX2_RENDER_TRACE")) { static std::atomic traces{}; diff --git a/src/video_core/shader_environment.h b/src/video_core/shader_environment.h index a1f1418599..dbb1ea4f7b 100644 --- a/src/video_core/shader_environment.h +++ b/src/video_core/shader_environment.h @@ -99,13 +99,6 @@ class GenericEnvironment : public Shader::Environment { /// kind than a cache record holds [[nodiscard]] bool CanBeSerialized() const noexcept; - /// The texture descriptor pool was flushed on the GPU thread before this environment's - /// translation moved to another thread, which may not flush: descriptors are read as they - /// are in memory. Nothing can make the pool dirty again while the GPU thread waits for it. - void SetTexturePoolFlushed() noexcept { - texture_pool_flushed = true; - } - [[nodiscard]] u64 CalculateHash() const; void Dump(u64 pipeline_hash, u64 shader_hash) override; @@ -163,7 +156,6 @@ class GenericEnvironment : public Shader::Environment { bool has_unbound_instructions = false; bool has_hle_engine_state = false; - bool texture_pool_flushed = false; }; class GraphicsEnvironment final : public GenericEnvironment { diff --git a/src/video_core/texture_cache/sampled_views.h b/src/video_core/texture_cache/sampled_views.h index 02a8280c05..12b48a9835 100644 --- a/src/video_core/texture_cache/sampled_views.h +++ b/src/video_core/texture_cache/sampled_views.h @@ -12,12 +12,13 @@ namespace VideoCommon { -/// Whether TextureCache::PrepareImageView prepares the view's image. Neither the null view, whose -/// image has no valid format and stays CpuModified (uploading it ends in UNREACHABLE, #904), nor a -/// buffer view, which has no image, is prepared. +/// Whether TextureCache::PrepareImageView prepares the view's image. Neither a view of the null +/// image (including empty descriptors with their own view ids), whose format is invalid and which +/// stays CpuModified, nor a buffer view is prepared. template [[nodiscard]] bool PreparesImage(const ImageViews& image_views, ImageViewId id) noexcept { - return id && id != NULL_IMAGE_VIEW_ID && !image_views[id].IsBuffer(); + return id && id != NULL_IMAGE_VIEW_ID && !image_views[id].IsBuffer() && + image_views[id].image_id != NULL_IMAGE_ID; } /// Whether TextureCache::PrepareImage would change a sampled image: upload it from guest memory diff --git a/src/video_core/texture_cache/texture_cache.h b/src/video_core/texture_cache/texture_cache.h index 397616a8d0..f23ebf676b 100644 --- a/src/video_core/texture_cache/texture_cache.h +++ b/src/video_core/texture_cache/texture_cache.h @@ -28,6 +28,7 @@ #include "video_core/dirty_flags.h" #include "video_core/nx2_fast_draw.h" #include "video_core/engines/kepler_compute.h" +#include "video_core/gpu_wait_stats.h" #include "video_core/guest_memory.h" #include "video_core/host1x/gpu_device_memory_manager.h" #include "video_core/nx2.h" @@ -59,6 +60,7 @@ using namespace Common::Literals; template TextureCache

::TextureCache(Runtime& runtime_, Tegra::MaxwellDeviceMemoryManager& device_memory_) : runtime{runtime_}, device_memory{device_memory_} { + Tegra::Texture::PrepareUnswizzleThreads(); debug_stats = std::getenv("EDEN_NX2_TC_LOG") != nullptr && Common::Log::IsLevelEnabled(Common::Log::Class::HW_GPU, Common::Log::Level::Debug); @@ -466,6 +468,16 @@ void TextureCache

::TickFrame() { total_used_memory -= (std::min)(total_used_memory, bytes); } total_used_memory -= (std::min)(total_used_memory, gpu_sentenced_bytes); + // Images the Vulkan recycler keeps are free memory to the collector: counted, a larger + // pool pushed it into high priority (more deletes and write-backs). With them past the + // critical memory, the pool gives back half of what it keeps each frame. + if constexpr (requires { runtime.GetRecycledImageMemory(); }) { + const u64 kept = runtime.GetRecycledImageMemory(); + total_used_memory -= (std::min)(total_used_memory, kept); + if (kept != 0 && total_used_memory + kept >= critical_memory) { + runtime.ShrinkRecycledImages(kept / 2); + } + } } // EDEN_NX2_MEM_TRACE=1: every 2 s, what the cache holds next to what the process and the // driver report (memory climbs). @@ -1310,6 +1322,7 @@ void TextureCache

::DownloadMemory(DAddr cpu_addr, size_t size) { auto copies = FullDownloadCopies(image.info); image.DownloadMemory(map, copies); pending_bytes += image.unswizzled_size_bytes; + GpuWaitStats::AddDownloadBytes(image.unswizzled_size_bytes); pending.push_back(Pending{image_id, map, std::move(copies)}); } flush_pending(); diff --git a/src/video_core/texture_cache/texture_cache_base.h b/src/video_core/texture_cache/texture_cache_base.h index 384b0c1dd6..87f317edfd 100644 --- a/src/video_core/texture_cache/texture_cache_base.h +++ b/src/video_core/texture_cache/texture_cache_base.h @@ -128,9 +128,11 @@ class TextureCacheChannelInfo : public ChannelInfo { DescriptorTable compute_image_table; DescriptorTable compute_sampler_table; - // TODO: still relies on bad iterators :( - std::unordered_map image_views; - std::unordered_map samplers; + // Flat maps, like the other hot texture cache maps: VisitImageView holds the try_emplace + // iterator across CreateImageView, which only erases from image_views (erasing never + // rehashes, and never removes the new entry: its id is still 0) and never inserts. + ::Common::unordered_map image_views; + ::Common::unordered_map samplers; TextureCacheGPUMap* gpu_page_table = nullptr; TextureCacheGPUMap* sparse_page_table = nullptr; diff --git a/src/video_core/textures/decoders.cpp b/src/video_core/textures/decoders.cpp index ef50be156b..2683eab325 100644 --- a/src/video_core/textures/decoders.cpp +++ b/src/video_core/textures/decoders.cpp @@ -5,19 +5,153 @@ // SPDX-License-Identifier: GPL-2.0-or-later #include +#include +#include #include +#include #include +#include +#include +#include #include +#include +#include #include "common/alignment.h" #include "common/assert.h" #include "common/bit_util.h" #include "common/div_ceil.h" +#include "common/logging.h" +#include "common/cpu_features.h" +#include "common/thread.h" #include "video_core/gpu.h" #include "video_core/textures/decoders.h" namespace Tegra::Texture { namespace { + +/// Large unswizzles (new textures while running through Ocarina of Time's Kokiri Forest: ~11% of +/// the split GPU back thread) are split by lines over helper threads plus the caller, which waits +/// for all of them: the result is the same, only sooner. A quarter of the host threads, at most 4 +/// (more competed with the guest cores and made busy spots drop harder). +class LinePool { +public: + static LinePool& Get() { + static LinePool pool; + return pool; + } + + [[nodiscard]] bool Enabled() const { + return !workers.empty(); + } + + /// Calls func(begin, end) over [0, count) in chunks; returns once every chunk is done. + void Run(u32 count, const std::function& func) { + std::unique_lock busy{run_mutex, std::try_to_lock}; + if (!busy || workers.empty()) { + func(0, count); + return; + } + { + std::scoped_lock lock{mutex}; + job = &func; + total = count; + chunk = (std::max)(8U, count / static_cast((workers.size() + 1) * 4)); + next.store(0, std::memory_order_relaxed); + pending = static_cast(workers.size()); + ++generation; + } + work_cv.notify_all(); + Work(func); + std::unique_lock lock{mutex}; + done_cv.wait(lock, [this] { return pending == 0; }); + job = nullptr; + } + +private: + /// At most 4 and a quarter of the host threads, and only the physical cores past the 6 the + /// guest cores and the GPU threads keep busy: on a 6-core Ryzen 5 5500, 3 workers competed + /// with them (no gain, slower movies). EDEN_NX2_UNSWIZZLE_THREADS sets the count (tests). + static u32 WorkerCount() { + if (const char* const text = std::getenv("EDEN_NX2_UNSWIZZLE_THREADS")) { + return (std::min)(static_cast(std::max(0, std::atoi(text))), 8U); + } + const u32 logical = std::thread::hardware_concurrency(); + u32 physical = logical / 2; + if (const auto cores = Common::GetProcessorCount(); cores && *cores > 0) { + physical = static_cast(*cores); + } + const u32 spare = physical > 6 ? physical - 6 : 0; + return (std::min)({4U, logical / 4, spare}); + } + + LinePool() { + const u32 count = WorkerCount(); + for (u32 i = 0; i < count; ++i) { + workers.emplace_back([this](std::stop_token stop) { Loop(stop); }); + } + LOG_INFO(HW_GPU, "Unswizzle helper threads: {}", count); + } + + ~LinePool() { + { + // Under the mutex: a worker between its predicate check and its wait would otherwise + // miss the notification and never return to be joined. + std::scoped_lock lock{mutex}; + for (auto& worker : workers) { + worker.request_stop(); + } + } + work_cv.notify_all(); + workers.clear(); + } + + void Work(const std::function& func) { + for (;;) { + const u32 begin = next.fetch_add(chunk, std::memory_order_relaxed); + if (begin >= total) { + return; + } + func(begin, (std::min)(begin + chunk, total)); + } + } + + void Loop(std::stop_token stop) { + Common::SetCurrentThreadName("Unswizzle"); + u64 seen = 0; + for (;;) { + const std::function* func; + { + std::unique_lock lock{mutex}; + work_cv.wait(lock, [&] { return stop.stop_requested() || generation != seen; }); + if (stop.stop_requested()) { + return; + } + seen = generation; + func = job; + } + Work(*func); + { + std::scoped_lock lock{mutex}; + --pending; + } + done_cv.notify_one(); + } + } + + std::mutex run_mutex; + std::mutex mutex; + std::condition_variable work_cv; + std::condition_variable done_cv; + const std::function* job{}; + u32 total{}; + u32 chunk{1}; + std::atomic next{}; + u32 pending{}; + u64 generation{}; + std::vector workers; +}; + constexpr u32 NativeGobOffset(u32 offset, bool native_tiling) { if (!native_tiling) return offset; const u32 difference = ((offset >> 5) ^ (offset >> 6)) & 1; @@ -63,11 +197,15 @@ void SwizzleImpl(std::span output, std::span input, u32 width, u32 const u32 block_depth_mask = (1U << block_depth) - 1; const u32 x_shift = GOB_SIZE_SHIFT + block_height + block_depth; - for (u32 slice = 0; slice < depth; ++slice) { + // Each (slice, line) writes its own bytes of the output: lines can go to other threads. + const auto lines = [&](u32 first, u32 end) { + for (u32 index = first; index < end; ++index) { + const u32 slice = index / height; + const u32 line = index % height; const u32 z = slice + origin_z; const u32 offset_z = (z >> block_depth) * slice_size + ((z & block_depth_mask) << (GOB_SIZE_SHIFT + block_height)); - for (u32 line = 0; line < height; ++line) { + { const u32 y = line + origin_y; const u32 swizzled_y = pdep(y); @@ -94,6 +232,15 @@ void SwizzleImpl(std::span output, std::span input, u32 width, u32 } } } + }; + const u32 total_lines = depth * height; + // From 1 MiB: splitting medium textures too (256 KiB) made busy spots drop harder. + constexpr u64 min_bytes = u64{1} << 20; + if (!TO_LINEAR && LinePool::Get().Enabled() && u64{pitch} * total_lines >= min_bytes) { + LinePool::Get().Run(total_lines, lines); + } else { + lines(0, total_lines); + } } template @@ -184,6 +331,10 @@ void Swizzle(std::span output, std::span input, u32 bytes_per_pixe } // Anonymous namespace +void PrepareUnswizzleThreads() { + [[maybe_unused]] const bool enabled = LinePool::Get().Enabled(); +} + void UnswizzleTexture(std::span output, std::span input, u32 bytes_per_pixel, u32 width, u32 height, u32 depth, u32 block_height, u32 block_depth, u32 stride_alignment, bool native_tiling) { diff --git a/src/video_core/textures/decoders.h b/src/video_core/textures/decoders.h index 6151967869..843c5bdf43 100644 --- a/src/video_core/textures/decoders.h +++ b/src/video_core/textures/decoders.h @@ -27,6 +27,9 @@ constexpr u32 SWIZZLE_X_BITS = 0b100101111; constexpr u32 SWIZZLE_Y_BITS = 0b011010000; /// Unswizzles a block linear texture into linear memory. +/// Starts the threads large unswizzles are split over now, instead of at the first one. +void PrepareUnswizzleThreads(); + void UnswizzleTexture(std::span output, std::span input, u32 bytes_per_pixel, u32 width, u32 height, u32 depth, u32 block_height, u32 block_depth, u32 stride_alignment = 1, bool native_tiling = false); diff --git a/src/video_core/video_core.cpp b/src/video_core/video_core.cpp index 4dc43c1f91..07ec49f6ac 100644 --- a/src/video_core/video_core.cpp +++ b/src/video_core/video_core.cpp @@ -11,12 +11,19 @@ #include "core/core.h" #include "core/frontend/emu_window.h" #include "core/frontend/graphics_context.h" +#include "video_core/buffer_cache/uniform_stream_reuse.h" #include "video_core/host1x/gpu_device_memory_manager.h" #include "video_core/host1x/host1x.h" -#include "video_core/nx2_async_readback.h" #include "video_core/nx2_cmdpage_diag.h" #include "video_core/nx2_download_trace.h" +#include "video_core/nx2_drain_check.h" +#include "video_core/nx2_entry_check.h" +#include "video_core/nx2_list_indirect.h" +#include "video_core/nx2_list_indirect_stats.h" +#include "video_core/nx2_list_stats.h" +#include "video_core/nx2_perf.h" #include "video_core/nx2_profile.h" +#include "video_core/nx2_read_trace.h" #include "video_core/nx2_stutter_trace.h" #include "video_core/renderer_base.h" #include "video_core/renderer_null/renderer_null.h" @@ -57,7 +64,15 @@ void CreateGPU(std::optional& gpu, Core::Frontend::EmuWindow& emu_wi VideoCommon::StutterTrace::StartSession(); Tegra::Nx2CmdPageDiag::ReadSwitch(); VideoCommon::Nx2DownloadTrace::ReadSwitch(); - VideoCore::AsyncReadback::ReadSwitch(); + VideoCommon::Nx2DrainCheck::ReadSwitch(); + VideoCommon::ReadTrace::ReadSwitch(); + Tegra::Nx2ListStats::ReadSwitches(); + VideoCommon::Nx2AcquireStats::ReadSwitch(); + VideoCommon::Nx2Perf::ReadSwitch(); + Tegra::Nx2EntryCheck::ReadSwitch(); + Tegra::Nx2ListIndirect::ReadSwitch(); + Tegra::Nx2ListIndirect::Stats::ReadSwitch(); + VideoCommon::ReadUniformStreamReuseSwitch(); const auto nvdec_value = Settings::values.nvdec_emulation.GetValue(); const bool use_nvdec = nvdec_value != Settings::NvdecEmulation::Off; diff --git a/src/video_core/vulkan_common/vulkan_device.cpp b/src/video_core/vulkan_common/vulkan_device.cpp index 12e6a0c5ba..61b243c38d 100644 --- a/src/video_core/vulkan_common/vulkan_device.cpp +++ b/src/video_core/vulkan_common/vulkan_device.cpp @@ -27,6 +27,7 @@ #include #include "common/settings.h" #include "common/settings_enums.h" +#include "video_core/nx2_read_trace.h" #include "video_core/nx2_stutter_trace.h" #include "video_core/vulkan_common/nsight_aftermath_tracker.h" #include "video_core/vulkan_common/vma.h" @@ -837,7 +838,9 @@ Device::Device(VkInstance instance_, vk::PhysicalDevice physical_, VkSurfaceKHR functions.vkAllocateMemory = TimedAllocateMemory; functions.vkFreeMemory = TimedFreeMemory; - VmaAllocatorCreateFlags flags = VMA_ALLOCATOR_CREATE_EXTERNALLY_SYNCHRONIZED_BIT; + // Synchronized inside VMA: the image recycler destroys dropped images on a thread of its own + // while the GPU thread allocates (vulkan_wrapper.cpp). + VmaAllocatorCreateFlags flags = 0; if (extensions.memory_budget) { flags |= VMA_ALLOCATOR_CREATE_EXT_MEMORY_BUDGET_BIT; } @@ -877,6 +880,7 @@ Device::Device(VkInstance instance_, vk::PhysicalDevice physical_, VkSurfaceKHR }; vk::Check(vmaCreateAllocator(&allocator_info, &allocator)); + vk::ImageRecycler::Open(allocator); owns_static_pipeline_cache = surface != VkSurfaceKHR{}; LoadStaticPipelineCache(); @@ -900,6 +904,7 @@ Device::~Device() { } } } + vk::ImageRecycler::Clear(allocator); vmaDestroyAllocator(allocator); } @@ -1260,6 +1265,15 @@ bool Device::GetSuitability(bool requires_swapchain) { FOR_EACH_VK_FEATURE_EXT(FEATURE_EXTENSION); FOR_EACH_VK_EXTENSION(EXTENSION); + // TEST (EDEN_NX2_READ_TRACE): the device's clock, read from the host (Scheduler::TraceRead). + if (VideoCommon::ReadTrace::Enabled()) { + if (supported_extensions.contains(VK_KHR_CALIBRATED_TIMESTAMPS_EXTENSION_NAME)) { + loaded_extensions.insert(VK_KHR_CALIBRATED_TIMESTAMPS_EXTENSION_NAME); + } else if (supported_extensions.contains(VK_EXT_CALIBRATED_TIMESTAMPS_EXTENSION_NAME)) { + loaded_extensions.insert(VK_EXT_CALIBRATED_TIMESTAMPS_EXTENSION_NAME); + } + } + extensions.depth_stencil_resolve = extensions.depth_stencil_resolve && (instance_version >= VK_API_VERSION_1_2 || extensions.create_renderpass2); @@ -1976,6 +1990,23 @@ void Device::ReleaseCustomBorderColorSamplers(size_t count) const { } u64 Device::GetDeviceMemoryUsage() const { + // The buffer and texture caches ask every frame; with the budget extension each answer is a + // driver and OS query (~0.1-0.5 ms on Windows): answers younger than 50 ms are reused. + constexpr s64 reuse_ns = 50'000'000; + const s64 now = std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()) + .count(); + const s64 last = memory_usage_time.load(std::memory_order_acquire); + if (last != 0 && now - last < reuse_ns) { + return memory_usage_value.load(std::memory_order_relaxed); + } + const u64 usage = QueryDeviceMemoryUsage(); + memory_usage_value.store(usage, std::memory_order_relaxed); + memory_usage_time.store(now, std::memory_order_release); + return usage; +} + +u64 Device::QueryDeviceMemoryUsage() const { VkPhysicalDeviceMemoryBudgetPropertiesEXT budget{}; budget.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MEMORY_BUDGET_PROPERTIES_EXT; physical.GetMemoryProperties(&budget); diff --git a/src/video_core/vulkan_common/vulkan_device.h b/src/video_core/vulkan_common/vulkan_device.h index 79bd94d977..9cba6be627 100644 --- a/src/video_core/vulkan_common/vulkan_device.h +++ b/src/video_core/vulkan_common/vulkan_device.h @@ -1085,7 +1085,9 @@ FN_MAX_LIMIT_LIST return extensions.memory_budget; } + /// Device memory in use, reused for 50 ms (QueryDeviceMemoryUsage asks the driver). u64 GetDeviceMemoryUsage() const; + u64 QueryDeviceMemoryUsage() const; /// The driver's current memory budget for this process (VK_EXT_memory_budget), in /// GetDeviceMemoryUsage() terms: the usage at which the main device-local heap reaches its @@ -1407,6 +1409,8 @@ FN_MAX_LIMIT_LIST bool dynamic_state3_alpha_to_one{}; bool supports_conditional_barriers{}; ///< Allows barriers in conditional control flow. mutable std::atomic custom_border_color_samplers_used{}; + mutable std::atomic memory_usage_value{}; ///< Last GetDeviceMemoryUsage() answer + mutable std::atomic memory_usage_time{}; ///< When it was taken (steady clock ns) u64 device_access_memory{}; ///< Total size of device local memory in bytes. u32 sets_per_pool{}; ///< Sets per Description Pool NvidiaArchitecture nvidia_arch{NvidiaArchitecture::Arch_AmpereOrNewer}; diff --git a/src/video_core/vulkan_common/vulkan_memory_allocator.cpp b/src/video_core/vulkan_common/vulkan_memory_allocator.cpp index eb129f4a43..a33668b7a5 100644 --- a/src/video_core/vulkan_common/vulkan_memory_allocator.cpp +++ b/src/video_core/vulkan_common/vulkan_memory_allocator.cpp @@ -256,6 +256,15 @@ VkResult MemoryAllocator::AllocateImage(const VkImageCreateInfo &ci, bool dedica VkImage handle{}; VmaAllocation allocation{}; VmaAllocationInfo alloc_info{}; + // An image of the same create info released before is taken from the recycler instead + // (vk::ImageRecycler). + const u64 recycle_key = vk::ImageRecycler::KeyOf(ci, dedicated); + if (recycle_key != 0 && vk::ImageRecycler::Take(recycle_key, allocator, handle, allocation)) { + image = vk::Image(handle, ci.usage, *device.GetLogical(), allocator, allocation, + device.GetDispatchLoader()); + image.SetRecycleKey(recycle_key); + return VK_SUCCESS; + } const u64 texels = u64{ci.extent.width} * ci.extent.height * ci.extent.depth * ci.arrayLayers; const bool huge = texels > MaxBackedImageTexels; VkResult result = huge ? VK_ERROR_OUT_OF_DEVICE_MEMORY @@ -335,6 +344,12 @@ VkResult MemoryAllocator::AllocateImage(const VkImageCreateInfo &ci, bool dedica image = vk::Image(handle, ci.usage, *device.GetLogical(), allocator, allocation, device.GetDispatchLoader()); + // Images placed in host memory under memory pressure are freed rather than kept. + if (recycle_key != 0 && alloc_info.memoryType < properties.memoryTypeCount && + (properties.memoryTypes[alloc_info.memoryType].propertyFlags & + VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT) != 0) { + image.SetRecycleKey(recycle_key); + } return VK_SUCCESS; } diff --git a/src/video_core/vulkan_common/vulkan_wrapper.cpp b/src/video_core/vulkan_common/vulkan_wrapper.cpp index 8975a105aa..55d04f96b7 100644 --- a/src/video_core/vulkan_common/vulkan_wrapper.cpp +++ b/src/video_core/vulkan_common/vulkan_wrapper.cpp @@ -5,13 +5,18 @@ // SPDX-License-Identifier: GPL-2.0-or-later #include +#include +#include #include +#include #include +#include #include #include #include "common/common_types.h" #include "common/logging.h" +#include "common/thread.h" #include "video_core/vulkan_common/vk_enum_string_helper.h" #include "video_core/vulkan_common/vma.h" #include "video_core/vulkan_common/vulkan_wrapper.h" @@ -530,10 +535,240 @@ void Image::SetObjectNameEXT(const char* name) const { void Image::Release() const noexcept { if (handle) { + if (recycle_key != 0 && allocation != nullptr && + ImageRecycler::Put(recycle_key, allocator, handle, allocation)) { + return; + } vmaDestroyImage(allocator, handle, allocation); } } +namespace { +struct RecycledImage { + u64 key; + VmaAllocator allocator; + VkImage handle; + VmaAllocation allocation; + u64 size; + std::chrono::steady_clock::time_point kept; +}; + +struct RecyclerState { + std::mutex mutex; + std::vector images; // oldest first + u64 bytes = 0; + VmaAllocator closed{}; ///< Torn down: keeps nothing more of it +}; + +RecyclerState& Recycler() { + static RecyclerState state; + return state; +} + +/// What the recycler keeps at most, and how long a kept image waits to be taken again. +constexpr u64 RecyclerCap = u64{512} << 20; +constexpr std::chrono::seconds RecycleKeepTime{10}; + +/// Images dropped from the recycler are destroyed on a thread of their own: freeing dedicated +/// memory is a kernel call (~0.2 ms), which the GPU thread would make otherwise. +struct Graveyard { + std::mutex mutex; + std::condition_variable cv; + std::vector images; + std::mutex destroy_mutex; ///< Held while a batch is destroyed (Clear waits for it) + bool started = false; +}; + +Graveyard& Reaper() { + // Never destroyed: its thread may still wait on it while the process exits. + static Graveyard* const graveyard = new Graveyard; + return *graveyard; +} + +void ReaperLoop(Graveyard& graveyard) { + std::vector batch; + while (true) { + std::unique_lock destroy_lock{graveyard.destroy_mutex, std::defer_lock}; + { + std::unique_lock lock{graveyard.mutex}; + graveyard.cv.wait(lock, [&graveyard] { return !graveyard.images.empty(); }); + // Taken with the batch, so Clear sees either the images or a batch in progress. + destroy_lock.lock(); + batch.swap(graveyard.images); + } + for (const RecycledImage& image : batch) { + vmaDestroyImage(image.allocator, image.handle, image.allocation); + } + batch.clear(); + } +} + +void Bury(std::vector&& images) { + Graveyard& graveyard = Reaper(); + { + std::scoped_lock lock{graveyard.mutex}; + if (!graveyard.started) { + graveyard.started = true; + std::thread{[&graveyard] { + Common::SetCurrentThreadName("VulkanReaper"); + ReaperLoop(graveyard); + }}.detach(); + } + graveyard.images.insert(graveyard.images.end(), images.begin(), images.end()); + } + graveyard.cv.notify_one(); +} + +/// Drops kept images that are too old, or the oldest until `room` more bytes fit the cap. +void TrimRecycler(RecyclerState& state, u64 room, std::chrono::steady_clock::time_point now) { + size_t drop = 0; + u64 bytes = state.bytes; + while (drop < state.images.size() && + (now - state.images[drop].kept > RecycleKeepTime || bytes + room > RecyclerCap)) { + bytes -= state.images[drop].size; + ++drop; + } + if (drop == 0) { + return; + } + const auto first = state.images.begin(); + const auto last = first + static_cast(drop); + Bury(std::vector(first, last)); + state.images.erase(first, last); + state.bytes = bytes; +} +} // Anonymous namespace + +u64 ImageRecycler::KeyOf(const VkImageCreateInfo& ci, bool dedicated) noexcept { + constexpr u64 Mul = 0x100000001b3ULL; + u64 key = 0xcbf29ce484222325ULL; + const auto mix = [&key](u64 value) { key = (key ^ value) * Mul; }; + mix(ci.flags); + mix(ci.imageType); + mix(ci.format); + mix((u64{ci.extent.width} << 32) | ci.extent.height); + mix(ci.extent.depth); + mix((u64{ci.mipLevels} << 32) | ci.arrayLayers); + mix(ci.samples); + mix(ci.tiling); + mix(ci.usage); + mix(ci.sharingMode); + mix(ci.initialLayout); + mix(dedicated ? 1 : 0); + if (ci.sharingMode == VK_SHARING_MODE_CONCURRENT) { + for (u32 index = 0; index < ci.queueFamilyIndexCount; ++index) { + mix(ci.pQueueFamilyIndices[index]); + } + } + for (auto* next = static_cast(ci.pNext); next != nullptr; + next = next->pNext) { + if (next->sType != VK_STRUCTURE_TYPE_IMAGE_FORMAT_LIST_CREATE_INFO) { + return 0; + } + const auto* list = reinterpret_cast(next); + mix(list->viewFormatCount); + for (u32 index = 0; index < list->viewFormatCount; ++index) { + mix(list->pViewFormats[index]); + } + } + return key == 0 ? 1 : key; +} + +bool ImageRecycler::Put(u64 key, VmaAllocator allocator, VkImage handle, + VmaAllocation allocation) noexcept { + VmaAllocationInfo info{}; + vmaGetAllocationInfo(allocator, allocation, &info); + if (info.size > RecyclerCap) { + return false; + } + RecyclerState& state = Recycler(); + std::scoped_lock lock{state.mutex}; + if (state.closed == allocator) { + return false; + } + const auto now = std::chrono::steady_clock::now(); + TrimRecycler(state, info.size, now); + state.images.push_back({key, allocator, handle, allocation, info.size, now}); + state.bytes += info.size; + return true; +} + +bool ImageRecycler::Take(u64 key, VmaAllocator allocator, VkImage& handle, + VmaAllocation& allocation) noexcept { + if (key == 0) { + return false; + } + RecyclerState& state = Recycler(); + std::scoped_lock lock{state.mutex}; + // Looked for before old images are dropped: after a few seconds without new images (the + // camera standing still), the first images of the next scene found theirs dropped just then. + bool found = false; + for (size_t index = state.images.size(); index-- > 0;) { + const RecycledImage& image = state.images[index]; + if (image.key == key && image.allocator == allocator) { + handle = image.handle; + allocation = image.allocation; + state.bytes -= image.size; + state.images.erase(state.images.begin() + static_cast(index)); + found = true; + break; + } + } + TrimRecycler(state, 0, std::chrono::steady_clock::now()); + return found; +} + +u64 ImageRecycler::KeptBytes() noexcept { + RecyclerState& state = Recycler(); + std::scoped_lock lock{state.mutex}; + return state.bytes; +} + +void ImageRecycler::Shrink(u64 bytes) noexcept { + RecyclerState& state = Recycler(); + std::scoped_lock lock{state.mutex}; + if (state.bytes <= bytes) { + return; + } + // Kept bytes never pass the cap, so here bytes < cap: dropping the oldest until + // `cap - bytes` more fit leaves at most `bytes`. + TrimRecycler(state, RecyclerCap - bytes, std::chrono::steady_clock::now()); +} + +void ImageRecycler::Open(VmaAllocator allocator) noexcept { + RecyclerState& state = Recycler(); + std::scoped_lock lock{state.mutex}; + if (state.closed == allocator) { + state.closed = nullptr; + } +} + +void ImageRecycler::Clear(VmaAllocator allocator) noexcept { + RecyclerState& state = Recycler(); + std::scoped_lock lock{state.mutex}; + state.closed = allocator; + std::erase_if(state.images, [&state, allocator](const RecycledImage& image) { + if (image.allocator != allocator) { + return false; + } + vmaDestroyImage(image.allocator, image.handle, image.allocation); + state.bytes -= image.size; + return true; + }); + // And the dropped ones the reaper has not destroyed yet, before the allocator goes. + Graveyard& graveyard = Reaper(); + std::scoped_lock graveyard_lock{graveyard.mutex}; + std::erase_if(graveyard.images, [allocator](const RecycledImage& image) { + if (image.allocator != allocator) { + return false; + } + vmaDestroyImage(image.allocator, image.handle, image.allocation); + return true; + }); + // A batch being destroyed: wait for it. + std::scoped_lock destroy_lock{graveyard.destroy_mutex}; +} + Image::MemoryBlock Image::GetMemoryBlock() const noexcept { if (allocation == nullptr) { return {}; diff --git a/src/video_core/vulkan_common/vulkan_wrapper.h b/src/video_core/vulkan_common/vulkan_wrapper.h index d99dbbbe3f..388bf483ec 100644 --- a/src/video_core/vulkan_common/vulkan_wrapper.h +++ b/src/video_core/vulkan_common/vulkan_wrapper.h @@ -710,7 +710,8 @@ class Image { Image(Image&& rhs) noexcept : handle{std::exchange(rhs.handle, VkImage{})}, usage{rhs.usage}, owner{rhs.owner}, - allocator{rhs.allocator}, allocation{rhs.allocation}, dld{rhs.dld} {} + allocator{rhs.allocator}, allocation{rhs.allocation}, dld{rhs.dld}, + recycle_key{rhs.recycle_key} {} Image& operator=(Image&& rhs) noexcept { Release(); @@ -720,9 +721,16 @@ class Image { allocator = rhs.allocator; allocation = rhs.allocation; dld = rhs.dld; + recycle_key = rhs.recycle_key; return *this; } + /// Released into ImageRecycler (for an image of the same create info) instead of + /// destroyed; 0 for images that are not recycled. + void SetRecycleKey(u64 key) noexcept { + recycle_key = key; + } + ~Image() noexcept { Release(); } @@ -764,6 +772,32 @@ class Image { VmaAllocator allocator = nullptr; VmaAllocation allocation = nullptr; const DeviceDispatch* dld = nullptr; + u64 recycle_key = 0; +}; + +/// Images on device-local memory, kept after release so an image of the same create info takes +/// one instead of allocating: Ocarina of Time makes and drops ~8 render targets a frame (its +/// per-frame allocator moves them, its shadow map changes size), and making, binding and freeing +/// each took ~0.2 ms of the GPU thread. Up to 512 MiB, for 10 seconds. +class ImageRecycler { +public: + /// Key of a create info and its allocation request, or 0 when it cannot be recycled (an + /// unknown pNext structure). + [[nodiscard]] static u64 KeyOf(const VkImageCreateInfo& ci, bool dedicated) noexcept; + /// Keeps the image for a later Take; false when it is not kept (the caller destroys it). + static bool Put(u64 key, VmaAllocator allocator, VkImage handle, + VmaAllocation allocation) noexcept; + /// An image kept under this key, if any. + static bool Take(u64 key, VmaAllocator allocator, VkImage& handle, + VmaAllocation& allocation) noexcept; + /// Destroys every kept image of this allocator and keeps nothing more (allocator teardown). + static void Clear(VmaAllocator allocator) noexcept; + /// Keeps images of a new allocator again, though it has the address of a cleared one. + static void Open(VmaAllocator allocator) noexcept; + /// Device memory the kept images hold (free memory to the texture cache's collector). + [[nodiscard]] static u64 KeptBytes() noexcept; + /// Drops the oldest kept images until at most `bytes` are kept. + static void Shrink(u64 bytes) noexcept; }; struct MemoryLocation {