Support inline hooking on Linux and macOS (x86-64) - #7
Draft
flobernd wants to merge 25 commits into
Draft
Conversation
…stall migration The accessor field on ZyrexOperation defers writing the user's "original" pointer until commit or revert, while all other threads are suspended, so a concurrent caller can never observe the patched jump and the original pointer disagreeing. This closes a recursion race where a caller re-entered the target through a not-yet-published trampoline pointer. Migration.cpp is reworked to validate concurrent-install migration in the one regime that's currently safe: install the hook while worker threads hammer the target (exercising ZyrexUpdateAllThreads' suspend-and-migrate path), then quiesce the workers before removing the hook. Removing a hook while callers are still active can free a trampoline a caller still references; a safe reclamation policy for that case is a separate, deferred design decision and is documented as such at the top of the file rather than exercised here.
ZyrexBarrierSystemShutdown released the TLS slot but not the calling thread's context vector. The per-thread cleanup callback only runs on thread exit, so repeated initialize/shutdown cycles on a long-lived thread leaked one context per cycle. Free the current thread's context before freeing the slot. Also broaden the barrier tests to cover recursion-depth counting, ZyrexBarrierTryEnterEx's max-depth gate, ZyrexBarrierGetRecursionDepth, handle independence, and leaving a handle with no active context.
A relocated relative CALL leaves a return address pointing into the trampoline, which instruction-pointer migration cannot fix up, so CALL in a prologue was rejected pending a trampoline-release policy. Introduce that policy: a removed hook's trampoline is quarantined by default (kept mapped, never reused) so a thread still executing in it or returning into it stays safe. ZyrexRemoveInlineHookEx exposes an opt-in ZYREX_REMOVE_HOOK_FLAG_RELEASE_TRAMPOLINE for callers that can guarantee no such reference; ZyrexShutdownEx adds a flag to release every remaining trampoline (including quarantined ones) at finalization. With that in place, relocate an in-range relative CALL like any other relative branch and reject only a target that no longer fits a rel32. Also reset the trampoline translation map at the start of relocation: a reused chunk still held the previous hook's entries, overflowing the map once the new release policy shifted the allocation pattern.
flobernd
force-pushed
the
feature/multi-platform-inline-hooks
branch
from
July 18, 2026 20:53
7e983cf to
ba66e77
Compare
The pinned dependencies/zycore commit tracked upstream master, which lacks the virtual-memory and thread-control primitives the POSIX trampoline and thread migration paths depend on, so CI could not build. Pin it to the zycore-c commit that adds those APIs so the build resolves and CI can run. Revert to a released zycore commit once the API additions land on master.
flobernd
force-pushed
the
feature/multi-platform-inline-hooks
branch
from
July 18, 2026 21:00
70ed6b0 to
35b13cc
Compare
CMake 4.0 removed compatibility with cmake_minimum_required below 3.5. The top-level project and the pinned zydis submodule both still declare 3.1, so configure fails on runners that ship CMake 4.x. Raise the policy floor across the dependency tree via the configure step's environment. It is set through env rather than a -D argument because the Windows runner's shell truncated the -D value to "3", which CMake rejects as malformed.
flobernd
force-pushed
the
feature/multi-platform-inline-hooks
branch
from
July 18, 2026 21:16
ba59d65 to
f3fb914
Compare
CallPrologueTarget is defined only by a GNU x64 inline-asm stub, but the g_call_original global and CallPrologueCallback helper referenced it unconditionally, so the MSVC build failed to link TestInlineHook with an unresolved external. The test that uses them already skips outside GNU x64, so move both helpers inside the same guard as the stub.
On MSVC in Release the identity targets HookTarget, ReleaseTarget and ShutdownReleaseTarget were proved to return their argument, so a direct call was folded to a constant and the tests could not observe the installed hook's redirection: they asserted the hooked value 0x1338 but always saw the original 0x1337. Route each target's result through a volatile so the compiler must emit a real call and use its actual return value. The functions stay identity, so GCC and Clang are unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends Zyrex inline hooking beyond Windows to Linux and macOS on x86-64, and completes the transactional install/revert path so hooks apply, revert, and coexist with concurrent installs on all three platforms.
Intention
Inline hooking was Windows-only: trampoline allocation, thread migration, and the commit path depended on Win32 primitives, and several transaction operations silently no-opped or leaked on POSIX. This makes the engine work end-to-end on Linux and macOS while keeping the Windows behaviour intact.
Changes
ZyrexRemoveInlineHookExandZyrexShutdownExflags.Testing
Full ctest suite green on Linux - relocation, trampoline, barrier, end-to-end inline hook, and concurrent thread migration - and ASan clean.
Related
Depends on zyantific/zycore-c#101 (virtual-memory and thread-control primitives).
Notes
Draft: the pinned zycore submodule (dd2211a) predates the new memory and thread APIs, so CI here will fail until zyantific/zycore-c#101 is merged and the submodule pointer is bumped. The Windows and macOS build paths are covered by CI but were validated end-to-end on Linux only.