fix: defer removed hook callback drops

[Jvlegod]

Hello maintainers,

I noticed issues with the current hook mechanism and have therefore made the following modifications.

This mechanism introduces a nesting counter parameter and briefly retains removed hooks until it is confirmed that the simulation has concluded.

Note: a similar method already exists in the unicorn core.

see:

• https://github.com/unicorn-engine/unicorn/blob/7c5db94191defc1e04a4f66f4eb1220903cba837/include/uc_priv.h#L363
• https://github.com/unicorn-engine/unicorn/blob/7c5db94191defc1e04a4f66f4eb1220903cba837/include/uc_priv.h#L418

[PhilippTakacs]

I'm not get what problem you want to solve. As you noticed unicorn does this already, so why do it again in the rust bindings?

[Jvlegod]

As you noticed unicorn does this already, so why do it again in the rust bindings?

unicorns do exist in this way.

However, in the rust bindings:

Previously, when remove_hook was called, the Rust callback was immediately freed; however, unicorn might still use it during nested execution, potentially leading to access of already freed memory.

Therefore, the goal here is to ensure that both the rust callback and the unicorn hook are released at the same time.

more easily example:

let hook_id = Rc::new(Cell::new(None));

let hook = uc.add_code_hook(CODE_START, CODE_START, move |uc, _, _| {
    // remove callback's hook
    // The callback might be dropped early while it is executing,
    // even though the lifetime of the hook has not yet ended.
    uc.remove_hook(hook_id.get().unwrap()).unwrap();
    uc.emu_start(CODE_START + 1, CODE_START + 2, 0, 0).unwrap();
}).unwrap();

hook_id.set(Some(hook));

You can see here :> :

unicorn-engine-rs/crates/unicorn/src/lib.rs

Line 1188 in e8ca055

inner.hooks.retain(|(id, _)| id != &hook_id);

Original code: when a matching hook_id is found and removed, the callback in rust is immediately dropped.

What is your take on this issue? :>

Note: Also due to the delayed deallocation mechanism(In C API), unicorn's C core library still holds an internal user_data pointer referencing this memory, which has already been freed by rust(Maybe I think).

[PhilippTakacs]

unicorn might still use it during nested execution

This is a bug, when a hook is deleted it should not be called again.

Can you test if unicorn-engine/unicorn#2253 fixes your issue?

[Jvlegod]

Can you test if unicorn-engine/unicorn#2253 fixes your issue?

sure, I will reply here as soon as I have finished testing.