Skip to content

feat(flow): add structured YAML selectors and property assertions - #56

Merged
saikrishna321 merged 3 commits into
appclawhq:mainfrom
qmkCamel:agent/structured-yaml-selectors
Aug 15, 2026
Merged

feat(flow): add structured YAML selectors and property assertions#56
saikrishna321 merged 3 commits into
appclawhq:mainfrom
qmkCamel:agent/structured-yaml-selectors

Conversation

@qmkCamel

Copy link
Copy Markdown
Contributor

Summary

This PR adds a deterministic accessibility-tree lane to YAML flows while preserving the existing natural-language/vision lane:

  • select elements by exact/contains/regex text, id, accessibilityId, type, hint, or value
  • disambiguate by state, index, spatial relations, and accessibility-tree relations
  • use structured targets from tap, type, wait, swipe, scroll, and assertion steps
  • assert element presence, state, content, count, dimensions, and coordinates with actionable selector diagnostics
  • retain selector diagnostics in reports, including redaction for password values

Structured targets fail deterministically when the accessibility tree cannot resolve them; they do not silently fall back to vision. Existing semantic string targets remain backward compatible.

- tap:
    target:
      id: com.example:id/search
      enabled: true
      clickable: true

- assert:
    target:
      id: com.example:id/submit
      rightOf:
        id: com.example:id/editor
    properties:
      visible: true
      enabled: true
      width:
        gte: 120

Android stability fixes

Real-device validation exposed two Android-specific gaps that are included here:

  • keep stable resource-id nodes even when UiAutomator2 reports them as non-clickable or without accessible text
  • apply UiAutomator2 idle/selector/action timeouts through the post-session settings endpoint instead of invalid session capabilities

This makes id-only child containers addressable and prevents dynamic apps from waiting for an idle accessibility tree before each page-source read.

Compatibility

  • preserves the v2.2.0 anchored scrollAssert target/target-proximity behavior from feat: Feat/anchored scroll assert #55
  • adds a focused regression for scrollAssert using both a structured selector and an anchored target
  • keeps legacy string YAML syntax and semantic/vision execution unchanged

Documentation

  • README examples and feature overview
  • docs/structured-selectors.md selector/assertion reference and migration guidance
  • docs/runner-architecture.md Android hierarchy and post-session settings details
  • OpenSpec change fix-android-structured-selector-device-stability

Validation

  • npm run typecheck
  • npm test — 24 files, 448 tests passed
  • npm run format:check
  • openspec validate fix-android-structured-selector-device-stability --strict
  • focused flow compatibility suite — 44 tests passed
  • Android physical device, OnePlus 8T / Android 14 / Kuaishou com.smile.gifmaker: 8/8 steps passed; compound selectors, state/geometry assertions, and rightOf relation verified (20260811T091013-9f73ee)
  • iOS physical device, iPhone 16 Plus / iOS 26.2.1 / Kuaishou com.kwai.gif.qmk.edge: two 8/8 structured-flow runs passed before the v2.2.0 rebase (20260811T083033-e6c872, 20260811T083117-0a7d83)

The post-rebase range-diff retains the Android fix unchanged; the only integration delta is preserving both the upstream anchored-scroll arguments and the new structured selector, covered by the focused compatibility test above.

中文

变更:
- 为 YAML Flow 增加精确、包含和正则选择器,以及状态、索引、空间和树关系匹配。
- 将结构化目标接入点击、输入、等待、滚动和断言执行路径,并输出匹配元素与失败诊断。
- 增加属性、数量、值和尺寸断言,补齐报告展示、敏感值脱敏、文档和回归测试。

验证:
- npm run typecheck
- npm test(22 个测试文件,427 项测试)
- npm run format:check
- git diff --check

English

Changes:
- Add exact, contains, and regex YAML selectors with state, index, spatial, and tree-relation matching.
- Wire structured targets through tap, type, wait, scroll, and assertion execution with matched-element diagnostics.
- Add property, count, value, and dimension assertions plus report rendering, secret redaction, documentation, and regression coverage.

Verification:
- npm run typecheck
- npm test (22 files, 427 tests)
- npm run format:check
- git diff --check
变更:
- 保留带 resource-id 的无文本、不可点击 Android 节点,允许结构化 selector 唯一定位并按中心坐标点击
- Android session 创建后通过 driver settings 写入 idle、selector 与 action acknowledgment timeout,失败时清理 session
- 补充中文 OpenSpec、回归测试及 selector/runner 文档

验证:
- openspec validate fix-android-structured-selector-device-stability --strict
- npm run typecheck
- npm test(24 个测试文件,433 项通过)
- git diff --cached --check

fix(android): stabilize structured selectors and page-source reads

Changes:
- retain id-bearing Android nodes without labels or clickable state so structured selectors can resolve and tap them by center coordinates
- apply Android idle, selector, and action acknowledgment timeouts through driver settings after session creation, cleaning up on failure
- add Chinese OpenSpec artifacts, regression coverage, and selector/runner documentation

Verification:
- openspec validate fix-android-structured-selector-device-stability --strict
- npm run typecheck
- npm test (24 test files, 433 tests passed)
- git diff --cached --check
中文

变更:
- 覆盖结构化 scrollAssert 与上游锚点滚动参数同时存在的执行路径。
- 防止 rebase 冲突处理再次丢失 selector 或 target。

验证:
- npm run build -w @appclaw/core
- npx vitest run tests/flow/structured-selector-runtime.test.ts tests/flow/run-instruction-proximity.test.ts tests/flow/parse-yaml-flow.test.ts(44 项通过)

English

Changes:
- Cover execution when structured scrollAssert selectors coexist with upstream anchored-scroll arguments.
- Prevent future conflict resolution from dropping either selector or target data.

Verification:
- npm run build -w @appclaw/core
- npx vitest run tests/flow/structured-selector-runtime.test.ts tests/flow/run-instruction-proximity.test.ts tests/flow/parse-yaml-flow.test.ts (44 tests passed)
@qmkCamel
qmkCamel marked this pull request as ready for review August 11, 2026 09:21
@saikrishna321
saikrishna321 merged commit 10363cd into appclawhq:main Aug 15, 2026
7 checks passed
github-actions Bot pushed a commit that referenced this pull request Aug 15, 2026
## [2.3.0](v2.2.0...v2.3.0) (2026-08-15)

### Features

* **flow:** add structured YAML selectors and property assertions ([#56](#56)) ([10363cd](10363cd))
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 2.3.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants