Skip to content

feat(input): add keyboard sequences and consistent target activation - #177

Merged
nekomeowww merged 4 commits into
mainfrom
fix/target-input-consistency
Sep 8, 2026
Merged

feat(input): add keyboard sequences and consistent target activation#177
nekomeowww merged 4 commits into
mainfrom
fix/target-input-consistency

Conversation

@nekomeowww

@nekomeowww nekomeowww commented Sep 8, 2026

Copy link
Copy Markdown
Member

Summary

Keyboard commands advertised --target but rejected it, and activation did not establish an explicit recipient contract. This PR routes application/window keyboard input through shared driver preparation and PID-bound delivery, then adds the approved key/combination/sequence hierarchy on that same path.

  • All keyboard invoke commands use InputKeyboard, including untargeted text/paste and selected dry-runs. Local handlers use typed argument conversions; Runner dispatch decodes once. The old clipboard shortcut parser and duplicate legacy invoke branches are removed.
  • PressKey and PressKeys use InputKeyboard. Key combinations accept explicit physical keys, including modified navigation/function keys; each press can repeat 1–255 times with an explicit interval. Ordered requests also support Unicode text and clipboard paste.
  • Validate the complete action list before input. Bind the recipient once and recheck it before each action/repetition. Foreground preparation confirms application/exact-window focus; background policies do not activate or fall back globally. Selecting an app/window does not select a text control.
  • Keep policy on each action and observed window ownership inside the window target. Missing RPC targets fail; global input requires an explicit foreground target. Replace the branch-only SendTargetedKeyboardInput RPC with InputKeyboard and PressKeys.
  • Preserve InputActionResult and unverified submission semantics. Failures stop the sequence and retain completed actions, the failed action index, and completed repetitions. Runner gRPC details survive into Rust client errors and CLI/MCP failure_details.keyboard_progress.
  • Add input.keys and input.keyboard, repetition options on input.key, and invoke protocol decoding for key arrays. CLI/MCP target facts and selected keyboard dry-run routing use command definitions; a subprocess regression catches accidental local dry-run validation when a Runner was selected. System-wide media commands continue to forbid targets.

Follow-up to #174, based on #176. The keyboard hierarchy extension was approved during review of this PR.

Compatibility and limits

Updated consumers and Runners are required for the new RPCs. A live unmodified main Runner returns UNIMPLEMENTED/unsupported; there is no global-input compatibility fallback. Selected keyboard dry-runs now require the selected Runner even without a target.

Existing global RPC wire shapes remain unchanged, and legacy cmd+a strings are accepted. Character presses now use physical keys and follow the active keyboard layout/IME; use TypeText for Unicode or exact literal text. Independent down/up, holds, whole-sequence retries, and disconnect-as-cancellation are not provided.

The originally reported ineffective first Cmd+A remains deferred. Input submission does not prove control consumption. NetEaseMusic still showed some submissions without the expected control effect. The separate activation-check false negative was reproduced on an AppKit fixture: PID lookup could return nil for a live process, and workspace caches could miss a restarted app. Input preparation now uses kernel liveness and AX frontmost state. Application lookup retains observed identities across transient misses, but validates each candidate against the kernel executable path and rejects dead or ambiguous recipients; these changes do not claim to fix modifier/control consumption. The earlier completed playback workflow remains historical evidence; this hierarchy revision does not claim a new stable end-to-end NetEase playback pass.

LobeHub must preserve stdout on nonzero exit and callers using gRPC must retain status details. AUV operation failures retain their structured envelope; parser/selection/tracing setup failures retain the existing stderr boundary. Callers should independently verify effects and must not blindly replay a partially executed sequence.

Verification

  • cargo fmt --check, cargo check, cargo test, git diff --check, cargo run --quiet -- invoke --help
  • Focused Rust suites: 286 passed across driver-common, macOS driver, invoke, CLI, and Rust SDK; platform probes run separately
  • CLI subprocess regressions for selected keyboard dry-runs and untargeted text validation; native-boundary failure injection; live application exit/restart, stale-window, and owner-mismatch rejection
  • scripts/generate-swift-bridge; swift build in the macOS native package
  • cargo clippy --all-targets --all-features completed with existing repository warnings
  • Buf generation/breaking checks passed; changed input schema formats cleanly. Existing repository lint/format findings remain documented.
  • SDK typecheck and tests: 53 passed, 1 skipped
  • Live AppKit readback: 0123456789 → three local Deletes → Runner sequence of two Deletes, Unicode XY, and Return → 01234XY, one submission. Ten local/Runner negative probes left the control unchanged; background input preserved the observed foreground app. NetEaseMusic separately showed triple Delete and complete query replacement.

Contract and evidence notes · Keyboard execution records · Original target-input evidence

Live activation regressions passed 30 consecutive preparations and 10 preparations for each of two fixture instances across an exit/restart. A persistent Runner rejected the terminated instance with structured not_found and zero progress, then delivered to its replacement with independent control readback. Latest live AppKit checks independently confirmed local and Runner clipboard paste (0123456789 local runner), clipboard restoration, and unchanged control/clipboard state for both text dry-run commands on both routes. The fixture now installs an Edit/Paste responder action so clipboard probes exercise a real shortcut.

No version release is included.

@nekomeowww nekomeowww changed the title fix(input): unify targeted keyboard delivery and activation feat(input): add keyboard sequences and consistent target activation Sep 8, 2026
@nekomeowww
nekomeowww merged commit cb56256 into main Sep 8, 2026
8 checks passed
@nekomeowww
nekomeowww deleted the fix/target-input-consistency branch September 8, 2026 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant