feat(input): add keyboard sequences and consistent target activation - #177
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Keyboard commands advertised
--targetbut rejected it, and activation did not establish an explicit recipient contract. This PR routes application/window keyboard input through shared driver preparation and PID-bound delivery, then adds the approved key/combination/sequence hierarchy on that same path.InputKeyboard, including untargeted text/paste and selected dry-runs. Local handlers use typed argument conversions; Runner dispatch decodes once. The old clipboard shortcut parser and duplicate legacy invoke branches are removed.PressKeyandPressKeysuseInputKeyboard. Key combinations accept explicit physical keys, including modified navigation/function keys; each press can repeat 1–255 times with an explicit interval. Ordered requests also support Unicode text and clipboard paste.SendTargetedKeyboardInputRPC withInputKeyboardandPressKeys.InputActionResultand unverified submission semantics. Failures stop the sequence and retain completed actions, the failed action index, and completed repetitions. Runner gRPC details survive into Rust client errors and CLI/MCPfailure_details.keyboard_progress.input.keysandinput.keyboard, repetition options oninput.key, and invoke protocol decoding for key arrays. CLI/MCP target facts and selected keyboard dry-run routing use command definitions; a subprocess regression catches accidental local dry-run validation when a Runner was selected. System-wide media commands continue to forbid targets.Follow-up to #174, based on #176. The keyboard hierarchy extension was approved during review of this PR.
Compatibility and limits
Updated consumers and Runners are required for the new RPCs. A live unmodified main Runner returns UNIMPLEMENTED/
unsupported; there is no global-input compatibility fallback. Selected keyboard dry-runs now require the selected Runner even without a target.Existing global RPC wire shapes remain unchanged, and legacy
cmd+astrings are accepted. Character presses now use physical keys and follow the active keyboard layout/IME; useTypeTextfor Unicode or exact literal text. Independent down/up, holds, whole-sequence retries, and disconnect-as-cancellation are not provided.The originally reported ineffective first Cmd+A remains deferred. Input submission does not prove control consumption. NetEaseMusic still showed some submissions without the expected control effect. The separate activation-check false negative was reproduced on an AppKit fixture: PID lookup could return nil for a live process, and workspace caches could miss a restarted app. Input preparation now uses kernel liveness and AX frontmost state. Application lookup retains observed identities across transient misses, but validates each candidate against the kernel executable path and rejects dead or ambiguous recipients; these changes do not claim to fix modifier/control consumption. The earlier completed playback workflow remains historical evidence; this hierarchy revision does not claim a new stable end-to-end NetEase playback pass.
LobeHub must preserve stdout on nonzero exit and callers using gRPC must retain status details. AUV operation failures retain their structured envelope; parser/selection/tracing setup failures retain the existing stderr boundary. Callers should independently verify effects and must not blindly replay a partially executed sequence.
Verification
cargo fmt --check,cargo check,cargo test,git diff --check,cargo run --quiet -- invoke --helpscripts/generate-swift-bridge;swift buildin the macOS native packagecargo clippy --all-targets --all-featurescompleted with existing repository warnings0123456789→ three local Deletes → Runner sequence of two Deletes, UnicodeXY, and Return →01234XY, one submission. Ten local/Runner negative probes left the control unchanged; background input preserved the observed foreground app. NetEaseMusic separately showed triple Delete and complete query replacement.Contract and evidence notes · Keyboard execution records · Original target-input evidence
Live activation regressions passed 30 consecutive preparations and 10 preparations for each of two fixture instances across an exit/restart. A persistent Runner rejected the terminated instance with structured
not_foundand zero progress, then delivered to its replacement with independent control readback. Latest live AppKit checks independently confirmed local and Runner clipboard paste (0123456789 local runner), clipboard restoration, and unchanged control/clipboard state for both text dry-run commands on both routes. The fixture now installs an Edit/Paste responder action so clipboard probes exercise a real shortcut.No version release is included.