tap by ref/coordinates rejected by published server; tap(text:) hits background route in stacked screens

[ReniDelonzek]

Suggestion — tap targeting on stacked/overlay screens

Observed version: npm flutter-skill@0.9.36 (latest) + pub flutter_skill@0.9.36

---

Problem 1 (release bug): the published npm binary lags behind the source

The published MCP server rejects tap by coordinates (x/y) and by ref, returning:

-32603 Internal error: Invalid argument(s): Must provide key or text for tap

However, in the source of the same 0.9.36 tag these paths already exist:

• lib/src/cli/tool_handlers/bf_interaction.dart (case 'tap') handles x/y → tapAt/tap_at, and forwards ref to client.tap(key, text, ref).
• lib/src/drivers/flutter_driver.dart implements tap({key, text, ref}) and tapAt(x, y); its error message is "Must provide key, text, or ref for tap".

The message returned at runtime ("...key or text", without ref) does not exist anywhere in the 0.9.36 source → the artifact shipped on npm was compiled from a commit older than the ref/coordinate support.

Suggested action: rebuild and republish the npm server binary from the current source (or publish 0.9.37) so that tap(ref: ...), tap(x, y) and the corresponding tool schema (which should advertise ref/x/y) actually reach users. This alone already fixes the case below: tap(ref: "<id from inspect>") would be enough.

Problem 2 (targeting robustness): tap(text:) hits the background route

Even with text/key targeting, _findElementByText and _findElementByKey (in flutter_skill.dart) walk the entire tree from rootElement and return the first match. When a screen is pushed over another (Navigator/Overlay), the background route is visited before the foreground one — so the tap can hit an element that is invisible, behind the current screen.

Repro

• Screen B is pushed (push) on top of screen A; A stays mounted (a very common pattern).
• The same text label exists on both — e.g. a tab label on the foreground screen B that also appears as a list/subtitle Text on the background screen A.
• tap(text: "<label>") triggers the background occurrence (the tap lands at the background element's coordinates), not the visible control on the foreground screen.

Cause

visit() short-circuits on the first match (pre-order, background first):

static Element? _findElementByText(String text) {
  Element? found;
  void visit(Element element) {
    if (found != null) return;                 // <- takes the 1st (background route)
    final w = element.widget;
    if (w is Text && w.data == text) { found = element; return; }
    if (w is RichText && w.text.toPlainText() == text) { found = element; return; }
    element.visitChildren(visit);
  }
  visit(WidgetsBinding.instance.rootElement!);
  return found;
}

Proposed fix — prefer the topmost-route / frontmost element

Collect all matches and pick the one in front. In Overlay order the topmost route is visited last, so matches.last is already a better heuristic; ideally confirm with a hit-test at the candidate's center (discards anything covered):

static Element? _findElementByText(String text) {
  final matches = <Element>[];
  void visit(Element element) {
    final w = element.widget;
    if ((w is Text && w.data == text) ||
        (w is RichText && w.text.toPlainText() == text)) {
      matches.add(element);
    }
    element.visitChildren(visit);
  }
  final root = WidgetsBinding.instance.rootElement;
  if (root == null) return null;
  visit(root);
  if (matches.isEmpty) return null;
  return _frontmostVisible(matches) ?? matches.last; // top of the stack
}

Where _frontmostVisible hit-tests at renderObject.localToGlobal(size.center) and keeps the candidate whose hit-test result belongs to itself (i.e., not covered by another route/barrier). Apply the same logic to _findElementByKey.

A more explicit alternative: scope the search to the current route (ModalRoute.of / topmost Navigator) and only walk its subtree.

Problem 3 (enter_text on fields without a ValueKey)

enter_text(key:)/_findElementByKey only match ValueKey<String>. Design-system text fields that wrap the inner editable with an internal GlobalKey and only expose Semantics(identifier: ...) are unreachable by key — only the "focused field + empty key" fallback works.

Suggestion: allow resolving enter_text/tap by Semantics.identifier/label too, and/or, when a matched element is not editable, descend into the nearest descendant EditableText. This makes masked/wrapped fields addressable without the focus trick.

---

Suggested priority

1. Problem 1 (republish the binary with ref/coordinates) — unblocks everything, no app change.
2. Problem 2 (prefer the topmost route in text/key matching) — fixes the whole class of stacked screens.
3. Problem 3 (Semantics/EditableText in enter_text) — quality of life for design systems.