fix(computer-use): win32 input actions fail truthfully; targeted mouse-down presses - #5910
fix(computer-use): win32 input actions fail truthfully; targeted mouse-down presses#5910EvanProgramming wants to merge 1 commit into
Conversation
… mouse-down presses
The win32 backend returned {action_sent: true} even when the PowerShell
subprocess never delivered the input:
- User32 P/Invoke type was only loaded in the bootstrap process, but every
action spawns a fresh powershell.exe, so [User32]:: calls were undefined
there and failed silently.
- withUser32 ignored the subprocess exit code, so failures still reported
success.
- left_mouse_down({target}) dropped the LEFTDOWN press due to a
ternary/concatenation precedence bug (only SetCursorPos was sent).
- cursor_position also referenced User32 without loading it.
Fix: define the User32 type in every action's process (USER32_DEF), make
withUser32 throw on a nonzero exit, and rewrite left_mouse_down so a
targeted call both moves and presses. create() now accepts an injectable
runner so the backend is testable without powershell.exe; regression tests
cover type-in-process, the failure path, and move+press.
Closes Hmbown#5908. Relates to Hmbown#5896.
|
Thanks @EvanProgramming for taking the time to contribute. This repository is observing a maintainer-managed PR intake gate in dry-run mode, so this pull request is staying open. This note helps maintainers prepare the allowlist before any enforcement is considered. Please read |
|
Closing with credit — this bug (#5896) was also fixed independently and has just merged as #5903 (self-contained |
… exec }) Follow-up to the closed Hmbown#5910, per the maintainer's invitation after the bug itself was fixed and merged in Hmbown#5903. This PR is refactor-only: it keeps main's merged behavior and adds a cleaner test seam. Previously the win32 backend's PowerShell runner (ps/psOk/psJson) lived at module scope, so backend tests had to fake powershell.exe on PATH (or run on Windows). Move the runner helpers into create() as instance-local functions that default to the production runner but can be overridden with create({ exec: { run } }) — the same injection pattern the harmonyos backend already uses. tests/win32.test.mjs now exercises the win32 backend on any host through the injected runner: actions run without powershell, nonzero exits reject (fail-truthful), and targeted mouse-down is verified to move + press with the User32 type defined in-process. Verified on macOS with no Windows involved: node --test tests/win32.test.mjs (4/4) and tests/backends.test.mjs (10/10, no regression). Refs Hmbown#5903, Hmbown#5910. Signed-off-by: EvanProgramming <evangonggyf@gmail.com>
Summary
Fixes #5908 (relates to #5896). The Windows computer-use backend reported
{action_sent: true}even when the underlying PowerShell subprocess never delivered the input.Three root causes, all in
crates/tui/plugins/computer-use/src/backends/win32.mjs:create()only ranAdd-Typein a bootstrap process, but everyps()spawns a freshpowershell.exe, so[User32]::mouse_event/SendInput/GetCursorPoswere undefined there and the call failed silently.withUser32ignored the exit code. It just returnedps(...), so a nonzero exit still produced{action_sent: true}.left_mouse_downprecedence bug.target ? A : "" + Bput themouse_event(LEFTDOWN)in the no-target branch, so a targeted call only sentSetCursorPos.Changes
USER32_DEFprepends theUser32P/Invoke definition to every User32-backed action so each is self-contained in its own process.withUser32throws anExecErroron a nonzero exit — no more false success.left_mouse_downrewritten so a targeted call both moves the cursor and presses.cursor_positionnow routes throughwithUser32(loads the type, fails truthfully instead of lying about a position).create({ exec })accepts an injectable runner so the backend is testable on any host withoutpowershell.exe(mirrors the harmonyos backend's pattern).Tests
New
crates/tui/plugins/computer-use/tests/win32.test.mjs(runs on any host via a mocked runner):action_sent: true).left_mouse_down({ target })sends bothSetCursorPosandLEFTDOWN.cursor_positionloads the type in-process and never reports a fake position.Verified:
node --test tests/win32.test.mjs→ 4/4 pass;node --test tests/backends.test.mjs→ 10/10 pass (no regression).