Cross-architecture PE analysis (x64 host -> 32-bit target)

[praydog]

Summary

Adds cross-architecture PE analysis — a 64-bit process can now map and correctly analyze a 32-bit (PE32) image — plus a growable exhaustive_decode seen-table that removes the x86 max_size cap.

Target architecture becomes a runtime property of the analyzed module rather than a compile-time property of the host build.

Changes

Growable exhaustive_decode seen table (Scan.hpp)

• Seen-address set grows on demand instead of being sized up front, so the x86 max_size cap could be dropped.
• Grow-only per thread; footprint tracks the largest function a thread actually decodes.
• Adds an OOM latch and a large-corpus growth/rehash stress test.

Cross-architecture PE analysis (Module.*, Scan.*, RTTI.cpp)

• TargetArch enum + host_arch() / pointer_width() helpers.
• get_module_arch() detects a module's architecture from its PE optional-header magic; is_mapped_module() distinguishes images we mapped ourselves from loader-loaded ones.
• Magic-aware get_dll_imagebase() (ImageBase differs in offset/width between PE32 and PE32+). get_module_size / get_module_sections / map_view_of_pe already worked for PE32 (SizeOfImage and IMAGE_FIRST_SECTION are magic-independent).
• Decode/scan primitives take a single optional TargetArch arch = host_arch(): decode_one, get_insn_size, exhaustive_decode, linear_decode, collect_basic_blocks[_into], get_highest_contiguous_block, resolve_displacement, scan_opcode / scan_disasm / scan_mnemonic, scan_ptr / scan_ptr_noalign.
• The former compile-time #if KANANLIB_ARCH_X86_32 branches in resolve_displacement (absolute [disp32] and imm32 references) are now runtime arch == X86 checks, so an x64 host can resolve x86 references.
• Pointer-width-sensitive paths (aligned/unaligned pointer scan, heuristic function-bucket pointer table walk) stride and read at the target's width.
• Function-bounds and RTTI entry points resolve the module's architecture internally, so their signatures are unchanged.
• RTTI: loaded modules keep the original host discovery path verbatim; mapped images are walked at the target pointer width, parsing the complete-object-locator / TypeDescriptor directly and matching on the decorated name (the host CRT undecorator cannot safely run on a foreign or mapped type_info). find_vtables' sort reads the locator at the target width.

Design note

On Windows a SEC_IMAGE-mapped image is relocated by the loader, so its stored pointers are already real host addresses. No address translation layer is needed — only the target's instruction mode and pointer width. That keeps the change small: the cross-arch footprint is roughly +280 / −136 across the five library files.

Compatibility

x64 behavior is unchanged: every new parameter defaults to host_arch(), and loaded modules keep the original RTTI path. No call sites required updating.

Tests

New kananlib-cross-arch-test (Windows-only, like the other PE-mapping tests) builds a synthetic relocatable PE32 in-memory and asserts, from both host architectures:

• PE32 mapping, architecture detection, image base, size, and section layout
• 32-bit decode (mov eax,[disp32] is 5 bytes; differs from a host-mode decode on x64)
• absolute [disp32] displacement resolution to the mapped address
• linear / exhaustive decode and basic-block collection
• 4-byte-wide aligned and unaligned pointer scans
• heuristic function-bounds discovery via the 4-byte pointer table
• RTTI vtable discovery by decorated name, including two same-named vtables ordered by subobject offset

ctest: 25/25 on x64 and 25/25 on x86 (Win32).

[Copilot]

Pull request overview

This PR adds cross-architecture PE analysis so a 64-bit host can map and analyze a 32-bit (PE32) image correctly, and replaces the fixed-size exhaustive_decode seen-table with a grow-on-demand structure to remove the prior x86 max_size cap.

Changes:

• Introduces TargetArch + runtime decode/pointer-width selection across decode/scan primitives.
• Reworks exhaustive_decode’s seen tracking into a growable open-addressing detail::SeenSet with OOM latching and adds coverage/stress tests.
• Adds a Windows-only cross-arch integration test that builds and maps a synthetic relocatable PE32 and validates decode/scan/RTTI behavior.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/TestScanCoverage.cpp	Adds deterministic SeenSet unit tests and decode-growth stress coverage; extends RWX test helper to allow custom sizes.
test/TestCrossArch.cpp	New Windows-only synthetic PE32 cross-arch integration test for mapping, decode, scan, function bounds, and RTTI.
test/StressTest.cpp	Clarifies back-edge detection rationale in CFG loop-correctness stress test.
test/CMakeLists.txt	Adds the new test target and tweaks FetchContent declarations.
test/cmake.toml	Registers the new kananlib-cross-arch-test target and updates sample-dir definitions.
src/Scan.cpp	Plumbs TargetArch through scan/decode APIs and adjusts pointer-width-sensitive logic for x86 targets.
src/RTTI.cpp	Adds mapped-module RTTI walking at target pointer width and updates vtable sorting to read locator pointers at target width.
src/Module.cpp	Adds PE optional-header-magic-based get_module_arch() and mapped-module detection; fixes PE32 ImageBase reads on x64.
include/utility/Scan.hpp	Adds runtime decode_mode/decode_data, TargetArch parameters with defaults, and new growable detail::SeenSet.
include/utility/Module.hpp	Introduces TargetArch, host_arch(), pointer_width(), and module-arch/mapped-module APIs.

Comments suppressed due to low confidence (2)

test/TestCrossArch.cpp:144

• IMAGE_BASE_RELOCATION::SizeOfBlock should be DWORD-aligned. 22 (8-byte header + 7*2-byte entries) is not; add an IMAGE_REL_BASED_ABSOLUTE padding entry and set SizeOfBlock to 24 to keep the reloc data spec-compliant.

    auto* pointer_reloc = reinterpret_cast<IMAGE_BASE_RELOCATION*>(bytes.data() + 0x60C);
    pointer_reloc->VirtualAddress = 0x2000;
    pointer_reloc->SizeOfBlock = 22;
    auto* pointer_entries = reinterpret_cast<uint16_t*>(pointer_reloc + 1);
    pointer_entries[0] = IMAGE_REL_BASED_HIGHLOW << 12;             // ptr @ 0x2000

test/CMakeLists.txt:72

• Same issue as above for spdlog: FetchContent_Declare(spdlog SYSTEM ...) is incompatible with the current cmake_minimum_required(VERSION 3.15) and can break configuration.

message(STATUS "Fetching spdlog (v1.12.0)...")
FetchContent_Declare(spdlog SYSTEM
	GIT_REPOSITORY

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[praydog]

Thanks — I verified each point against the code and its history rather than taking them at face value. 5 of 6 were real and are fixed; 1 I'm declining here with reasoning. Both suppressed low-confidence comments were also valid and are fixed.

How the two behavioral claims were proven

Both are x86-host-only, which is why the existing suite never caught them. I added two discriminating tests and ran them on both builds before changing any production code:

	x64 build	x86 build
pre-fix	7/7 pass (host == X64 masks both)	2 failed
post-fix	7/7 pass	7/7 pass

Pre-fix x86 failures:

FAIL: get_module_arch(...) == host_arch()   TestCrossArch.cpp:348
FAIL: *as_x64 == start + 8                  TestCrossArch.cpp:368

Fixed

src/Module.cpp:138 — unknown optional-header magic classified as X64. Real. On an x86 host a ROM (0x107) or malformed magic flipped decode mode to 64-bit and pointer width to 8 — a regression against the original compile-time behavior, which always used the host arch. Now only the two defined magics select an arch; anything else falls back to host_arch(). Covered by test_module_arch_falls_back_on_unknown_magic.

src/Scan.cpp:359 — scan_ptr_noalign used host pointer width for X64. Real, and an asymmetry: the aligned scan_ptr already used an explicit uint64_t, while this path used sizeof(uintptr_t). Now explicit uint64_t. Zero change on x64 (sizeof(uintptr_t) == 8). Covered by test_scan_ptr_noalign_uses_target_width, whose fixture places a 4-byte-matching decoy before the real 8-byte value so a host-width search picks the wrong offset.

src/Scan.cpp:302 — ++it past one-past-the-end. Correct, it is UB. Worth noting it is pre-existing — origin/main's scan_ptr has the identical ++it after std::find returns end — and I carried it into the new template. Replaced with break; behaviorally identical, since the loop condition failed immediately either way. It is live code, reached on every not-found scan.

test/CMakeLists.txt:61,72 — FetchContent_Declare(... SYSTEM ...) vs cmake_minimum_required(3.15). Correct, and this PR did introduce it: it is unintended cmkr regeneration drift from a newer generator, unrelated to the feature. SYSTEM needs CMake ≥3.25. Reverted both to match origin/main. While there I also dropped the KANANLIB_SAMPLE_DIR wiring for this target — dead, since the test builds its PE32 in memory.

TestCrossArch.cpp:69,140,144 — reloc SizeOfBlock not DWORD-aligned. Correct per spec. To be precise: it was not a live bug — the block is last in the directory, so nothing downstream misaligned, and the loader processed all seven entries (every relocation-dependent assertion, including the new second vtable, passed). Still worth being spec-correct: added a trailing IMAGE_REL_BASED_ABSOLUTE padding entry, SizeOfBlock 22 → 24, directory/section size 34 → 36.

Declining, with reasoning

src/Scan.cpp:1930 — addr + ptr_width < region_end should be <=. The observation is correct: the read spans [addr, addr+ptr_width), so <= is the accurate bound and < skips one final in-bounds slot. But this is pre-existing and untouched by this PR — origin/main line 1932 reads:

for (auto addr = region_start; addr + sizeof(uintptr_t) < region_end; addr += sizeof(uintptr_t)) {

The only change here was substituting the width constant (sizeof(uintptr_t) → ptr_width), which is identical on both native builds. Changing < to <= would add one candidate pointer per readable region to x64 heuristic function discovery — a behavior change to the native path, which this PR explicitly guarantees it does not make. It is a reasonable fix, but it belongs in its own change where that delta can be evaluated on its own merits.

Verification

ctest 25/25 on x64 and 25/25 on x86 (Win32) after the fixes, plus the two new discriminating tests.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

[praydog]

Second review: 1 of 2 valid and fixed. (The scan_value_aligned type-punning/alignment comment is intentionally out of scope for this PR — see below.)

Fixed — src/Module.cpp:121, get_dll_imagebase on unknown magic

Valid, and it is the same class of defect as the get_module_arch one from the last round — I fixed that one and missed its sibling three lines above. Confirmed against origin/main, which read the field through the host-layout PIMAGE_NT_HEADERS:

return ntHeaders->OptionalHeader.ImageBase;   // origin/main

So for an unrecognized magic the behavior was:

host	original	this PR (before fix)
x64	offset 24, 8 bytes	offset 24, 8 bytes — unchanged
x86	offset 28, 4 bytes	offset 24, 8 bytes — regressed

Only the two defined magics now select a layout; anything else returns nullopt, which matches the DOS-signature and NT-signature checks the function already performs (there is no sensible default image base to guess, unlike get_module_arch, where falling back to the host arch is meaningful).

Proof — test_dll_imagebase_reads_by_magic pins all three cases: PE32 reads the 4-byte field at optional-header offset 28, PE32+ reads the 8-byte field at offset 24 (a wrong-layout read lands on the zeroed upper half, so this discriminates), and ROM (0x107) / garbage (0xDEAD) magics are rejected. Unlike last round's two regressions, this one is not host-specific, so it fails on both builds before the change:

pre-fix   x64: 7 passed, 1 failed    x86: 7 passed, 1 failed
          FAIL: !get_dll_imagebase(...).has_value()   TestCrossArch.cpp:393
post-fix  x64: 8 passed, 0 failed    x86: 8 passed, 0 failed

Note the PE32 and PE32+ assertions passed before the fix too — the layout selection was already correct; only the unknown-magic path was wrong.

On the "potentially fault when the optional header isn't large enough" part: reading Magic itself, and the 32-byte span either real layout requires, is the same exposure origin/main had, so I have not added SizeOfOptionalHeader validation here — that would be a broader hardening change with its own risk of rejecting real images, and it belongs in its own commit.

Out of scope — src/Scan.cpp:315, scan_value_aligned type punning

The observation is legitimate C++: casting arbitrary memory to T* and running std::find over it is not strictly conforming, and an unaligned start would be a real problem on an alignment-strict target.

I'm deliberately not changing it in this PR. This is a scan library whose entire premise is reinterpreting foreign process/image memory as typed values — the same pattern is pervasive in Scan.cpp and RTTI.cpp and predates this PR. scan_value_aligned is a faithful extraction of origin/main's scan_ptr loop, which did exactly this with uintptr_t*; the only thing this PR changed is which width is used. Converting it to memcpy-based loads would be a codebase-wide change to a hot scanning path, evaluated on its own merits, not folded into a cross-architecture feature. Both supported targets are x86/x64, where unaligned loads are well-defined at the hardware level.

Verification

ctest 25/25 on x64 and 25/25 on x86 (Win32), cross-arch suite 8/8 on both.