Skip to content

Prevent Castle Town guard-house crashes with a larger render-target pool - #1202

Merged
hckrman101 merged 1 commit into
NotRexed:mainfrom
nathanaelries:fix/guardhouse-render-target-pool
Nov 9, 2026
Merged

hckrman101 merged 1 commit into
NotRexed:mainfrom
nathanaelries:fix/guardhouse-render-target-pool

Conversation

@nathanaelries

Copy link
Copy Markdown
Contributor

Talking to the soldier in Castle Town’s guard house while rotating the camera repeatedly crashed at 2560×1440 with the 305 MiB render-target pool. Reserving 512 MiB resolved the reported reproduction.

This gives the pool a 512 MiB minimum while preserving existing 4K/ultrawide growth and explicit HOS_RT_POOL_MB overrides. The existing graphics-heap/budget hooks use the same sizing function, so the backing allocation grows with it. The base reservation increases by 207 MiB.

The initial stop was native NVN: texture handle ids -1/264. The companion handle correction matches the original SDK: sign-extend both IDs, combine them, and return zero when the texture ID is zero. That correction alone did not resolve the room: it exposed a subsequent allocator crash in f_8005ed00. Disabling write-back diff and the separate present queue also did not resolve it. Raising the pool to 512 MiB with those changes retained did.

Validation

  • python -I -B tests/run_all.py --jobs 1 --strict nvn rt_pool: 5 passed.
  • New production-function fixtures: 12 pool-sizing cases and 13 handle cases, passing with GCC and Clang. The pool regression fails against current main’s old sizing; the handle regression fails against the old constructor.
  • Local differential check: the corrected source routine and the patched native routine matched the original SDK in 10,196 boundary/random argument pairs. SDK bytes and generated artifacts stay local; the committed fixtures contain authored constants only.
  • In-game confirmation: v2.0.8 (source ed212f62), Linux, RX 7800 XT/RADV, docked 2560×1440 at 100% resolution, frame generation off. The same soldier/camera action passed after setting the pool to 512 MiB. Steam’s overlay remained enabled for the working controller setup.

The complete game has not been rebuilt from this exact branch, and Windows runtime behavior has not been tested. The pool recommendation is based on the repeatable allocation failure and the successful larger-pool test; peak pool use has not been instrumented. No game files, saves, settings files, binaries, caches, or SDK code are included.

@hckrman101 hckrman101 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks good to me. What I checked:

  • hos_rt_pool_size is the only sizing source for the pool config store, the Graphics heap growth, the module budget correction and the f_80b72340 heap trim in nx2/oot_patches.py. So the 512 MB floor applies to all four consistently. At 1440p those patches used to do nothing (growth was 0). Now they add 207 MB, which is less than the 382 MB the 4K path already uses, so it stays inside a configuration that already works.
  • The env override still wins, including values below the floor. 4K/ultrawide results are unchanged (687 / 1373). The 1080p output mode recently added on main gets 512, which is fine. It merges cleanly with that change.
  • I checked the handle change against the Vulkan side. An all-ones handle decodes to texture slot 0xfffff and sampler 0xfff. texture_of, sampler_of and texture_slot_version all bounds-check those, so it falls back to the dummy texture and default sampler instead of indexing out of range.
  • Both tests compile the real function bodies through the existing block() helper, they're picked up by run_all.py, and the main cases fail against the old code (die -> exit 86, and 305 instead of 512). I checked the expected numbers in the fixtures by hand.
  • No overlap with other open PRs (#1179 touches nn_applet.c, but in different hunks).

Two non-blocking notes are inline: (1) the floor is applied after scaling, so 4K doesn't get the same headroom the guard house needed at 1440p, and (2) a one-time log for negative texture ids would keep pool exhaustion visible now that the die() is gone.

Comment thread host/nn_applet.c
if (wide_px > px) px = wide_px;
const uint64_t v = e ? strtoull(e, NULL, 0) << 20 : px > 2560 * 1440 ? (game * px / (2560 * 1440) + 0xfffff) & ~0xfffffULL : game;
const uint64_t scaled = px > 2560 * 1440 ? (game * px / (2560 * 1440) + 0xfffff) & ~0xfffffULL : game;
const uint64_t minimum = 512ULL << 20;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something to consider: the floor is applied after scaling. If the guard house needs more than 305 MB at 1440p because of render targets, it would need roughly 2.25x that at 4K, but 4K still gets 687 MB (305 scaled). Applying the floor to the base first (game < minimum ? minimum : game, then scale) would give 4K the same relative headroom, at about +465 MB at 4K. Not tested at 4K either way, so I'm fine leaving it as is for now. It's just worth knowing if a 4K report of the same room comes in.

Comment thread host/nvn_native.c
/* The SDK sign-extends both ids before combining them, and texture 0 always means no handle.
A missing texture can have id -1 (Castle Town's guard house): it stays the all-ones sentinel,
rather than stopping the game here. Keep the signed ids, including their upper bits. */
RET(t ? (uint64_t)t | (uint64_t)s << 20 : 0);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Matching the SDK makes sense. One small thing: the old die("texture handle ids -1/...") was the only clear sign that the render-target pool had run out (issue 1031 was diagnosed from it, and the comment above hos_rt_pool_size still names it as the symptom). Maybe a one-time hos_log when t < 0, so a future pool exhaustion shows up in the log before the allocator crash in f_8005ed00?

@hckrman101
hckrman101 merged commit 73b9732 into NotRexed:main Nov 9, 2026
@hckrman101

Copy link
Copy Markdown
Collaborator

Feel free to open follow-up

notquiteog added a commit to notquiteog/Hookshot that referenced this pull request Nov 9, 2026
Microphone input, motion controls, the FPX sticky rescue, the guard-house
render-target pool and upstream's test fixes. vk_menu.h keeps both the
neural-rendering options and the microphone's; the rumble and output-resolution
tests take upstream's fixtures, with the render-target expectation brought up to
NotRexed#1202's 512 MB floor (upstream's own copy was red on that).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants