diff --git a/PR_DESCRIPTION.md b/PR_DESCRIPTION.md new file mode 100644 index 0000000000..e1110a3b33 --- /dev/null +++ b/PR_DESCRIPTION.md @@ -0,0 +1,124 @@ +# [Design] Self-Healing Locators for Datagrok UI Tests + +> **Status:** Design proposal. No executable code in this PR. +> **Goal:** Get reviewer alignment on architecture, fingerprint schema, prompt +> contract, healing policy, and rollout plan **before** any code is written. + +## TL;DR + +UI tests in Datagrok (package tests on Puppeteer, plus a growing Playwright E2E +suite) break when the DOM, CSS, or labels shift. This proposal introduces a +two-phase **self-healing locator** system: + +- **Runtime fallback** (library, no LLM): when a selector misses, a + deterministic resolver walks a 4-tier priority hierarchy (semantic IDs → + stable structural → text content → visual/positional) using a + **fingerprint** captured during green test runs. A confidence score gates + the outcome: high → heal silently, mid → heal + flag for review, low → + fail the test. +- **Offline maintenance** (CLI, uses Claude API): low-confidence and + unresolved cases land in a queue. A separate tool prompts Claude with the + fingerprint + accessibility tree, validates each proposed selector against + a live page in headless Puppeteer/Playwright, and opens a follow-up PR + with codemod'd test source updates. + +The two phases share one fingerprint schema, one confidence model, and one +audit log format. The LLM is the **last resort**, not the first move. + +## What's in this PR (design only) + +``` +libraries/self-healing-locators/ + README.md + docs/ + 01-design.md [FULL] + 02-fingerprint-spec.md [FULL] + 03-confidence-model.md [FULL] + 04-runtime-flow.md [FULL] + 05-offline-flow.md [FULL] + 06-datagrok-integration.md [COMPACT] + 07-policy.md [FULL] + 08-review-process.md [COMPACT] + 09-eval-and-rollout.md [COMPACT] + prompts/ + system.md [FULL] + user-template.md [FULL] + examples/ + 01-testid-rename.md + 02-aria-label-change.md + 03-structural-refactor.md + schemas/ + fingerprint.schema.json + healing-event.schema.json + tool-response.schema.json + api/ + healing-api.md [FULL] + +tools/self-healing-cli/ + README.md + docs/ + cli-overview.md [COMPACT] +``` + +## Decisions baked into this proposal + +| Decision | Choice | Where it's discussed | +|---|---|---| +| Project location | `libraries/self-healing-locators` + `tools/self-healing-cli` | `01-design.md` | +| Healing mode | Hybrid: runtime fallback + offline source updates | `04-runtime-flow.md`, `05-offline-flow.md` | +| LLM provider | Anthropic Claude API | `05-offline-flow.md` | +| Model strategy | Tiered: Haiku → Sonnet → Opus | `05-offline-flow.md` § "Model selection" | +| Fingerprint storage | Centralized JSON registry under `libraries/self-healing-locators/registry/` (not in this PR) | `02-fingerprint-spec.md` § "Storage" | +| Fingerprint capture | Passive (auto-update on green runs) + explicit (`healing.anchor()` API) | `api/healing-api.md` | +| Screenshots | Feature-flagged, off by default | `02-fingerprint-spec.md` § "Visual layer" | +| LLM in runtime | **No.** Runtime is deterministic and fast. | `04-runtime-flow.md` | +| Source code mutation | Offline only, via codemod, gated by PR review | `05-offline-flow.md` § "Codemod & PR" | + +## Review focus + +Please read in this order — each doc is short and self-contained: + +1. **`01-design.md`** — confirm overall shape and component boundaries +2. **`07-policy.md`** — confirm the "what we heal vs what we let fail" boundary +3. **`02-fingerprint-spec.md`** — confirm the schema before we lock it in +4. **`prompts/system.md` + `prompts/user-template.md`** — confirm the contract with Claude +5. **`03-confidence-model.md`** — confirm the weights and thresholds (these are calibratable, not hardcoded forever) +6. The rest — orientation only + +## Explicit non-goals + +- We do **not** propose changes to the Datagrok platform itself in this PR. + Any `getWidgetStatus()` extensions or new `data-testid` conventions are + noted as recommendations in `06-datagrok-integration.md`, to be addressed + separately. +- We do **not** heal anything beyond locators. Assertions, timing, navigation, + and test logic are out of scope. See `07-policy.md` for the rationale. +- We do **not** mutate test source files automatically without a PR. + +## Open questions (please flag in review) + +1. **Registry location** — should fingerprints live next to tests, in a + sibling directory, or in a single central registry? See + `02-fingerprint-spec.md` § "Storage" for trade-offs. +2. **Failure budget** — what's the acceptable number of LLM-suggested heals + per week before we treat it as a signal that the platform needs more + stable test IDs? +3. **PR ownership** — when the offline tool opens a PR, who is the default + reviewer? Test author? QA team? Code owners of the affected package? +4. **Cost ceiling** — concrete dollar limit per offline run before the tool + pauses and asks for human approval. + +## After this PR is approved + +- Iterate on review comments until docs are signed off. +- Open implementation PRs in this order: + 1. `libraries/self-healing-locators` core types + fingerprint capture + 2. Runtime resolver + confidence scorer (no LLM) + 3. Audit log + healing queue writer + 4. `tools/self-healing-cli` skeleton + Claude client + 5. Prompt + tool_use response validation + 6. Candidate validator (headless browser) + 7. Codemod + PR opener + 8. Eval harness + first 30-50 fixtures + +Each step is independently reviewable and shippable behind a feature flag. diff --git a/libraries/self-healing-locators/README.md b/libraries/self-healing-locators/README.md new file mode 100644 index 0000000000..d62a8384eb --- /dev/null +++ b/libraries/self-healing-locators/README.md @@ -0,0 +1,52 @@ +# @datagrok-libraries/self-healing-locators + +Resilient UI element discovery for Datagrok tests. When a primary selector +breaks because of a UI change, this library finds the same element through +alternative signals and reports its confidence in the match. + +## Status + +**Design phase.** This directory currently contains design docs, schemas, and +prompt specs only. No runtime code has been written yet. See +[`docs/01-design.md`](./docs/01-design.md) for the proposed architecture and +[`PR_DESCRIPTION.md`](../../PR_DESCRIPTION.md) at the repo root for the rollout +plan. + +## What this library does (when implemented) + +- Captures **fingerprints** of UI elements during green test runs — semantic + IDs, ARIA roles, stable classes, parent chain, visible text, optional + visual hash, and Datagrok widget metadata via `getWidgetStatus()`. +- At runtime, when a primary selector misses, walks a 4-tier priority + hierarchy (semantic → structural → text → visual) to relocate the + element, **without calling an LLM**. +- Computes a **confidence score** for the match. High confidence heals + silently; mid-confidence heals but flags for review; low confidence fails + the test and queues the case for offline analysis. +- Writes an **audit log** for every healing decision. + +The companion CLI in [`tools/self-healing-cli`](../../tools/self-healing-cli) +handles offline maintenance: it consumes the queue, calls Claude to propose +new selectors for hard cases, validates them against a live page, and opens +a PR with updated test sources. + +## What this library does **not** do + +- Does not call any LLM at runtime. Tests stay fast and offline-capable. +- Does not heal assertions, timing, or test logic. Locators only. +- Does not mutate test source files. Source edits happen via codemod in the + offline CLI, behind a PR. + +## Quick links + +- [Architecture overview](./docs/01-design.md) +- [Fingerprint schema](./docs/02-fingerprint-spec.md) +- [Confidence model](./docs/03-confidence-model.md) +- [Runtime flow](./docs/04-runtime-flow.md) +- [Offline flow](./docs/05-offline-flow.md) +- [Healing policy](./docs/07-policy.md) +- [Public API](./api/healing-api.md) + +## License + +Same as the parent repository. diff --git a/libraries/self-healing-locators/api/healing-api.md b/libraries/self-healing-locators/api/healing-api.md new file mode 100644 index 0000000000..54331afa2d --- /dev/null +++ b/libraries/self-healing-locators/api/healing-api.md @@ -0,0 +1,194 @@ +# Healing API + +> Status: full detail. This is the surface test authors see and depend on. + +## Module + +```ts +import * as healing from '@datagrok-libraries/self-healing-locators'; +``` + +The library is framework-agnostic at the type level. Concrete adapters +ship for Puppeteer (used by Datagrok package tests) and Playwright +(used by the standalone E2E suite). The core API works the same way +through either adapter. + +## Configuration + +Done once per test process, typically in a global setup file. + +```ts +healing.configure({ + registryPath: 'libraries/self-healing-locators/registry/v1', + auditLogPath: 'test-output/self-healing/locator-events.jsonl', + queuePath: 'test-output/self-healing/healing-queue.jsonl', + thresholds: { high: 0.80, mid: 0.55 }, + enableScreenshots: false, + enableTier4Visual: false, + primarySelectorTimeoutMs: 1000, + perTierBudgetMs: { tier1: 200, tier2: 300, tier3: 200, tier4: 500 }, + // Optional overrides; see docs/03-confidence-model.md for defaults + weights: undefined, +}); +``` + +`configure()` is idempotent. Calling it twice with identical options is +a no-op. Calling it with conflicting options throws — config is global +state, but it's not mutable global state. + +## Core: `resolve()` + +The drop-in replacement for `page.locator(selector)`. + +```ts +const element = await healing.resolve(page, '[data-testid="submit"]', { + anchorName?: string; // explicit name; auto-derived otherwise + action?: 'click' | 'type' | 'assert' | 'read' | 'hover'; + purpose?: 'normal' | 'destructive' | 'auth'; // policy hint +}); +``` + +Returns the framework's element handle (Puppeteer `ElementHandle`, +Playwright `Locator`). Callers use it normally — `await element.click()`, +etc. + +Behavior: + +1. Tries the primary selector (with the configured timeout). +2. On miss, runs the resolver chain. +3. Returns the resolved element on HIGH or MID; throws on LOW or fail. +4. Writes one audit event per call (success or fail). + +If the primary selector hits, `resolve()` is just a passive +fingerprint-refresh wrapper with negligible overhead. + +## Explicit anchors: `anchor()` + +Preferred for critical paths. Forces an explicit name and surfaces +metadata into the fingerprint. + +```ts +const submit = await healing.anchor(page, 'login-submit', { + selector: '[data-testid="submit"]', + action: 'click', + purpose: 'auth', +}); +await submit.click(); +``` + +Equivalent to `resolve()` with `anchorName: 'login-submit'`, but: + +- The `anchorName` is required, not derived. Renames go through a + deprecation path (see § "Renaming anchors"). +- The `purpose` field is recorded in the fingerprint and influences + the policy gates (e.g. `auth` requires explicit anchors; the policy + refuses to heal auto-derived anchors on auth flows). +- The library can attach optional fingerprint hints + (`hints.elementIsInsideViewer`) without the author having to compute + them. + +## Renaming anchors + +Anchor names are part of the contract between tests and the registry. +A rename is a deliberate refactor, not a silent fix. + +```ts +healing.deprecateAnchor('old-name', { until: '2026-09-01' }); +const submit = await healing.anchor(page, 'new-name', { ... }); +``` + +The library: + +- Loads both fingerprints; new anchor inherits from old until the + deadline. +- Writes a warning to the audit log on each use of the old name. +- After the deadline, the old anchor is pruned by `grok-heal registry prune`. + +## Reading the audit log + +For dashboards, custom reporters, or test-time introspection. + +```ts +const events = await healing.audit.read({ since: '2026-04-01' }); +const counters = healing.audit.snapshot(); +// counters: { total_resolves, primary_hits, tier1_hits, ..., failed } +``` + +The audit log is jsonl, append-only. The library exposes a streaming +reader so consumers do not have to parse it themselves. + +## Marking review decisions + +After a healing PR is reviewed, decisions feed back to the offline tool +so it doesn't re-propose the same heal next run. + +```ts +// Programmatic alternative to `grok-heal mark` +await healing.audit.markCase(caseId, { + decision: 'accepted' | 'rejected', + reason?: string, +}); +``` + +Programmatic marking is mostly for the CLI; humans typically use +`grok-heal mark` from the command line. + +## Error types + +```ts +class HealingError extends Error { + caseId: string; + reason: + | 'no_fingerprint' + | 'all_tiers_missed' + | 'ambiguous_candidates' + | 'low_confidence' + | 'platform_version_skew' + | 'registry_corrupt' + | 'unhealable_destructive' + | 'unhealable_auth'; + details: { /* per-reason fields */ }; +} +``` + +`HealingError` extends `Error` so existing test reporters render it +sensibly. The `caseId` lets a developer pull up the audit entry directly +from the failure message. + +## What the API never does + +- **Never returns null silently.** Either resolves to an element or + throws. There is no in-band "didn't find it" sentinel. +- **Never retries on transient failures.** That's the test framework's + job. Self-healing fixes locator drift, not flakiness. +- **Never logs to stdout.** All logging goes through the audit log or + a configurable logger callback. +- **Never makes network calls.** No metrics services, no telemetry, no + Anthropic SDK in this package. + +## Adapter packages + +The core library exposes interfaces; adapters implement them. + +``` +@datagrok-libraries/self-healing-locators // core +@datagrok-libraries/self-healing-locators-puppeteer // adapter +@datagrok-libraries/self-healing-locators-playwright // adapter +``` + +Adapters are thin: they translate between core types and the +framework's element/page types, and they implement the framework-specific +parts of fingerprint capture (taking a screenshot, reading the +accessibility tree, etc.). + +The Datagrok Puppeteer test framework uses the Puppeteer adapter. The +Playwright E2E suite uses the Playwright adapter. The core API surface +is identical from the test author's perspective. + +## API stability + +- `1.0` is the first stable release; signatures above ship as is. +- Pre-1.0 versions may evolve based on early-adopter feedback from the + volunteer package(s) in Stage 2 of the rollout. +- Breaking changes after 1.0 follow standard semver and require a + deprecation window of at least one minor version. diff --git a/libraries/self-healing-locators/docs/01-design.md b/libraries/self-healing-locators/docs/01-design.md new file mode 100644 index 0000000000..80093d48fa --- /dev/null +++ b/libraries/self-healing-locators/docs/01-design.md @@ -0,0 +1,300 @@ +# 01 — Design Overview + +> Status: full detail. This is one of the documents to read carefully. + +## 1. Problem statement + +Datagrok UI tests — both the package tests on the Puppeteer-based framework +and the growing Playwright E2E suite — fail when the DOM, CSS classes, ARIA +labels, or text content shift between releases. The failures are not real +regressions; they are locator drift. Today the cost of locator drift is paid +by humans who hand-edit tests after each platform or package release. + +The goal of this proposal is to absorb routine drift automatically while +preserving the ability of tests to fail loudly when something genuinely +broke in the product. + +## 2. Design principles + +These are the non-negotiables. Every detail downstream serves them. + +**P1. The LLM is the last resort, not the first move.** Most drift is +trivial: a class renamed, a `data-testid` updated, a wrapper added. A +deterministic resolver that walks a priority hierarchy resolves the vast +majority of cases at zero cost and with no nondeterminism. Only what cannot +be resolved deterministically gets escalated to Claude, and only offline. + +**P2. Real bugs must surface.** Self-healing must never mask a regression. +If a button is gone or its semantics changed, the test fails. The +confidence model is the gatekeeper here — it is computed from objective +matches against a fingerprint captured when the test last passed, not from +an LLM's self-reported confidence. + +**P3. Every heal is auditable.** Every decision — resolved by which tier, +what matched, what didn't, what the resulting confidence was, who or what +approved it — is written to a structured log. Without this, the system +becomes a black box that quietly hides failures. + +**P4. Source code mutation is gated by a human.** The offline tool produces +a PR. It does not commit directly to a branch that anything depends on. A +human reviews the diff and merges. + +**P5. Runtime is fast and offline-capable.** No network calls, no LLM, no +external services in the runtime path. Everything required at runtime ships +with the library and the fingerprint registry. + +**P6. The system is calibratable, not hardcoded.** Weights, thresholds, +model selection rules, and prompt templates are configuration, not code. +Teams can tune them without forking the library. + +## 3. High-level architecture + +``` + ┌──────────────────────────────┐ + │ Test Authoring / Recording │ + │ │ + │ Fingerprints captured │ + │ during green runs and via │ + │ explicit healing.anchor() │ + └──────────────┬───────────────┘ + │ + ▼ + ┌──────────────────────────────┐ + │ Fingerprint Registry │ + │ (versioned JSON files) │ + └──────────────┬───────────────┘ + │ + ┌───────────────────┴────────────────────┐ + │ │ + ▼ ▼ + ┌───────────────────────────┐ ┌──────────────────────────────┐ + │ RUNTIME (library) │ │ OFFLINE (CLI tool) │ + │ │ │ │ + │ ┌─────────────────────┐ │ │ ┌────────────────────────┐ │ + │ │ Locator Resolver │ │ │ │ Queue Reader │ │ + │ │ Tier 1: semantic │ │ │ │ (low-conf + failed │ │ + │ │ Tier 2: structural │ │ │ │ runtime cases) │ │ + │ │ Tier 3: text │ │ │ └──────────┬─────────────┘ │ + │ │ Tier 4: visual │ │ │ ▼ │ + │ └──────────┬──────────┘ │ │ ┌────────────────────────┐ │ + │ ▼ │ │ │ Prompt Builder │ │ + │ ┌─────────────────────┐ │ │ │ (fingerprint + a11y │ │ + │ │ Confidence Scorer │ │ │ │ tree + optional img) │ │ + │ └──────────┬──────────┘ │ │ └──────────┬─────────────┘ │ + │ ▼ │ │ ▼ │ + │ ┌─────────────────────┐ │ │ ┌────────────────────────┐ │ + │ │ Decision Gate │ │ │ │ Claude Client │ │ + │ │ HIGH → heal │ │ │ │ (Haiku → Sonnet → │ │ + │ │ MID → heal+flag │ │ │ │ Opus, tool_use) │ │ + │ │ LOW → fail+queue │ │ │ └──────────┬─────────────┘ │ + │ └──────────┬──────────┘ │ │ ▼ │ + │ ▼ │ │ ┌────────────────────────┐ │ + │ ┌─────────────────────┐ │ │ │ Candidate Validator │ │ + │ │ Audit Log Writer │ │ │ │ (headless browser, │ │ + │ └─────────────────────┘ │ │ │ re-scores via the │ │ + │ │ │ │ │ same scorer) │ │ + └─────────────┼─────────────┘ │ └──────────┬─────────────┘ │ + │ │ ▼ │ + │ │ ┌────────────────────────┐ │ + │ │ │ Codemod + PR Opener │ │ + │ │ └────────────────────────┘ │ + │ └──────────────┬───────────────┘ + │ │ + ▼ ▼ + ┌──────────────────────────────────────────────────────────────────┐ + │ Audit log (locator-events.jsonl) │ + │ Healing queue (healing-queue.jsonl) │ + │ Review dashboard (out of scope for this PR) │ + └──────────────────────────────────────────────────────────────────┘ +``` + +The two columns share three things: +1. **The fingerprint schema** (`schemas/fingerprint.schema.json`). +2. **The confidence scorer** — same code, same weights. The offline tool + re-scores Claude's proposals using the runtime scorer; Claude does not + get to vote on its own confidence. +3. **The audit log format** (`schemas/healing-event.schema.json`). + +## 4. Component boundaries + +### 4.1 `libraries/self-healing-locators` (runtime library) + +A `@datagrok-libraries/*` package consumed by: +- Datagrok package tests (Puppeteer-based) +- Standalone Playwright projects (E2E suite) + +Exports: +- `healing.anchor(name, element)` — explicit fingerprint capture +- `healing.resolve(selector, options)` — drop-in replacement for + `page.$(selector)` / `page.locator(selector)` that triggers the resolver + on miss +- `healing.configure(opts)` — thresholds, registry path, audit log path, + feature flags +- `healing.audit` — read-side API for tools and dashboards + +No dependency on the Anthropic SDK. No network code at all. + +See [`api/healing-api.md`](../api/healing-api.md) for the full surface. + +### 4.2 `tools/self-healing-cli` (offline CLI) + +A `tools/*` package, in line with `datagrok-tools`. Distributed as an npm +binary. Depends on: +- `@datagrok-libraries/self-healing-locators` (for the shared scorer, + schemas, and registry I/O) +- `@anthropic-ai/sdk` (Claude API client) +- `puppeteer` or `playwright` for candidate validation + +Commands (sketch — finalized in `tools/self-healing-cli/docs/cli-overview.md`): + +``` +grok-heal queue # show pending cases +grok-heal run [--model auto] # process the queue +grok-heal validate # dry-run a single case +grok-heal pr # open a PR with codemod'd changes +grok-heal eval # run the regression eval suite +``` + +The CLI is invoked from CI on a schedule (e.g. nightly) or manually after +a release that produced runtime-queued cases. + +### 4.3 Fingerprint registry + +A versioned directory (location decided in this PR — see +`02-fingerprint-spec.md` § "Storage") holding one JSON file per logical +test or per package, depending on the chosen layout. Committed to git. +Updates flow through normal review; the offline CLI does not bypass review. + +## 5. Data flow scenarios + +### Scenario A — Trivial drift, runtime resolves it + +1. Test calls `healing.resolve('[data-testid="submit"]')`. +2. Selector misses (the platform renamed the testid to `submit-btn`). +3. Resolver loads the fingerprint for that anchor. +4. Tier 1 finds an element matching the recorded `ariaLabel + ariaRole` + pair. The fallback `testId` candidate `submit-btn` also matches the + same element. +5. Scorer computes confidence ≥ HIGH. +6. Decision gate heals silently. Test continues. +7. Audit log records: `tier=1, confidence=0.92, matched=[ariaLabel, + ariaRole, testId-fuzzy], healed_selector='[data-testid="submit-btn"]'`. + +No LLM, no human involvement. Cost: a few milliseconds and one log line. + +### Scenario B — Structural change, runtime escalates + +1. The submit button is now nested inside a new `` wrapper + and its `data-testid` is gone. +2. Tier 1 misses. Tier 2 finds an element matching tag, stable classes, + and one of three parent-chain levels — but the chain is shifted by one. +3. Tier 3 matches on visible text "Submit". Tier 4 finds three elements + with similar bounding boxes; the right one is among them. +4. Scorer computes confidence in the MID band: 0.62. +5. Decision gate heals (test continues) but flags the case. +6. Audit log records `flag=review`. The case is also written to + `healing-queue.jsonl` for offline analysis. +7. Next nightly CLI run: Claude sees the fingerprint, the new a11y tree, + and proposes 3 candidate selectors. The validator confirms 1 of them + matches a single element with confidence ≥ HIGH under the runtime + scorer. +8. CLI opens a PR updating the test source to use the new selector. A + human reviews and merges. + +### Scenario C — Real bug, system fails loudly + +1. The submit button was removed from the page entirely (refactor mistake). +2. All four tiers miss or return only weak partial matches. +3. Confidence < LOW threshold. +4. Decision gate fails the test. The case is queued. +5. CLI runs, Claude proposes selectors that the validator finds either + missing or pointing to different semantic elements (e.g. a "Cancel" + button in the same position). +6. CLI marks the case `unable_to_heal` with reason + `no_semantic_match_in_dom`. No PR opened. The case is escalated to a + human via the review dashboard. + +This scenario is the entire point of P2. The system must produce this +outcome reliably. + +## 6. What we deliberately keep out + +- **In-test LLM calls.** Latency, cost, nondeterminism. Forbidden. +- **Auto-merge of healing PRs.** Even high-confidence offline heals get a + human reviewer, at least until the eval suite has earned trust. +- **A separate ML model.** A scoring formula with calibrated weights is + enough for the deterministic tiers. If we later need a learned model, + we can swap the scorer; the architecture supports it. +- **Healing of non-locator failures.** Timing flakes, network errors, + state-leak failures — different problems, different solutions. Mixing + them with locator healing dilutes both. +- **A custom format for fingerprints.** Plain JSON, schema-validated. No + protobuf, no SQLite, no embeddings (yet). + +## 7. What is calibratable vs hardcoded + +| Calibratable (config) | Hardcoded (code) | +|---|---| +| Tier weights | Tier order | +| Confidence thresholds (HIGH/MID/LOW) | Scoring formula shape | +| Model selection rules | tool_use response schema | +| Prompt templates | Audit log schema | +| Feature flags (screenshots, etc.) | Public API surface | +| Per-package overrides | Registry layout | + +Anything calibratable lives in `self-healing.config.json` next to the +registry. Defaults ship with the library. + +## 8. Dependencies on the platform + +We rely on three platform capabilities. Two exist; one is a recommendation. + +**Exists:** `Widget.getWidgetStatus()` returns a runtime structure intended +specifically for automated testing and introspection (per the JS API docs). +We use this in Tier 1 as a Datagrok-specific semantic anchor. + +**Exists:** the Inspector tool (`Alt + I`) exposes the registered widget +tree. The fingerprint capture utility taps into the same data source so +fingerprints reflect the platform's own model of the page. + +**Recommended (not blocking):** a convention for `data-testid` attributes +on common UI primitives (`ui.button`, `ui.input.*`, dialog headers, +ribbon items). Without this, semantic Tier 1 match is weaker and we lean +harder on text and structure. See +[`06-datagrok-integration.md`](./06-datagrok-integration.md) for the +specific suggestions. + +## 9. Failure modes we explicitly handle + +| Failure mode | Handling | +|---|---| +| Registry file missing | Treat as fresh capture; record for next green run | +| Registry file stale (schema version mismatch) | Migration script; fail closed if migration is ambiguous | +| All tiers miss with non-zero partial scores | Confidence < LOW → fail + queue | +| Multiple candidates tied at HIGH | Fail closed, queue for offline disambiguation. Never guess. | +| LLM API down | Offline CLI exits cleanly, queue persists | +| LLM proposes selector that matches multiple elements | Validator rejects, marks as `ambiguous_match` | +| LLM proposes selector that matches a different semantic element | Validator computes runtime score; if < HIGH, reject | +| Codemod conflicts with concurrent edits | Skip case, requeue, notify in PR description | + +## 10. Why this shape + +We considered three alternatives and rejected them: + +**A. Pure runtime LLM healing.** Rejected: violates P1, P5, and creates a +hard dependency on network availability for tests. Also: the audit story is +weak because LLM reasoning isn't reliably reproducible. + +**B. Pure offline LLM rewriting (no runtime resolver).** Rejected: every +trivial drift becomes a Claude call and a PR. Noise overwhelms signal, +costs balloon, and the team starts ignoring PRs. + +**C. Visual-only matching (image diff + perceptual hash).** Rejected as a +primary mechanism: too brittle for SPAs that re-render frequently, fails +on dynamic content, and offers no semantic explanation when it errs. +Visual signals stay as Tier 4 for cases where they genuinely help. + +The hybrid two-phase design absorbs trivial drift cheaply at runtime and +escalates only what is genuinely hard. The LLM does what it's good at +(reasoning over messy structure) without doing what it's bad at (being +the source of truth for production-critical decisions). diff --git a/libraries/self-healing-locators/docs/02-fingerprint-spec.md b/libraries/self-healing-locators/docs/02-fingerprint-spec.md new file mode 100644 index 0000000000..535c07c53a --- /dev/null +++ b/libraries/self-healing-locators/docs/02-fingerprint-spec.md @@ -0,0 +1,333 @@ +# 02 — Fingerprint Specification + +> Status: full detail. The fingerprint schema is the contract between +> capture, runtime resolution, and the offline LLM. Lock this carefully. + +## 1. What a fingerprint is + +A fingerprint is a structured snapshot of an element captured when a test +last passed. It contains enough orthogonal signals that, when the primary +selector breaks, we can re-identify the same element through a different +combination of signals — and quantify our confidence in the match. + +A fingerprint is not a CSS selector. It is the **evidence** we use to find +or generate a working selector. + +## 2. Schema (informal) + +The formal JSON Schema lives in +[`schemas/fingerprint.schema.json`](../schemas/fingerprint.schema.json). +Here is the human-readable form, with rationale per field. + +```ts +interface ElementFingerprint { + // Identity + schemaVersion: '1'; + anchorName: string; // logical name from healing.anchor() + // or auto-derived from selector + test + capturedAt: string; // ISO 8601 + capturedBy: 'auto' | 'explicit'; + testRef: { + file: string; // path relative to repo root + suite?: string; + test: string; + selectorAtCapture: string; // the selector that worked + }; + + // Tier 1 — semantic + semantic: { + testId?: string; // data-testid value, if any + domId?: string; // id, only if it passes stability filter + ariaLabel?: string; + ariaRole?: string; // button, dialog, tab, ... + name?: string; // accessible name (from a11y tree) + dgWidgetType?: string; // from getWidgetStatus(), e.g. 'Viewer' + dgViewerType?: string; // e.g. 'scatter-plot', 'grid' + }; + + // Tier 2 — stable structural + structural: { + tagName: string; + stableClasses: string[]; // post-filter (see § 4) + parentChain: ParentLink[]; // up to 3 levels, see below + indexInParent?: number; // last-resort sibling index + }; + + // Tier 3 — text content + text: { + visibleText?: string; // normalized, trimmed, length-capped + placeholder?: string; + title?: string; + valueAtCapture?: string; // for inputs, only if not sensitive + }; + + // Tier 4 — visual / positional + visual: { + boundingBox?: { // viewport-relative at capture + x: number; y: number; + width: number; height: number; + }; + viewportSize?: { w: number; h: number }; + visualHash?: string; // pHash; only when feature flag enabled + screenshotRef?: string; // file path under registry, optional + }; + + // Stability hints discovered at capture time + hints: { + domIdLooksStable: boolean; // see filter logic § 4 + classesAreCssInJs: boolean; + textIsLikelyDynamic: boolean; // detected via heuristics, e.g. "Hello, $user" + elementIsInsideViewer?: string; // viewer type if applicable + }; + + // For the LLM and for debugging + contextSnippet: { + accessibilityNode?: object; // a11y subtree at this element, depth 2 + htmlSnippet?: string; // sanitized outerHTML, truncated + }; +} + +interface ParentLink { + tagName: string; + testId?: string; + ariaRole?: string; + dgWidgetType?: string; + stableClasses?: string[]; // already filtered +} +``` + +## 3. Why these fields and not others + +Every field justifies its weight in the scoring formula. We omit: + +- **XPath of the element.** Too brittle, encodes everything that drifts. + We reconstruct an XPath at runtime if needed; we never store one as + ground truth. +- **Full outer HTML.** Heavy, leaky, encourages bad matching. We keep a + truncated snippet only as context for Claude in the offline phase. +- **Computed styles.** Volatile across themes and platform versions. +- **Event listeners.** Not observable reliably in headless contexts. +- **Embeddings.** Not yet. Adds infrastructure and explainability cost + without a clear win at our current failure rates. Revisit after the + eval suite has data. + +We include: + +- **`name` from the accessibility tree** even when it duplicates `ariaLabel` + or `visibleText`. The a11y tree resolves these per the platform's + algorithm, which is what assistive tech and Playwright `getByRole` use. + Storing it makes Tier 1 matches more robust. +- **`dgWidgetType` and `dgViewerType`** because Datagrok has a layer of + semantics above the DOM that most SPAs lack. Using it gives us a stable + anchor that survives many DOM-level changes. +- **`hints`** because they are computed once at capture and used by both + the runtime resolver and the offline prompt. Storing them avoids + recomputation and ensures the LLM and the resolver see the same view. + +## 4. Normalization rules + +At capture, raw values pass through normalizers. Both capture and +resolution use the same normalizers — the registry stores the **post- +normalization** form. Normalizers are part of the library's public API +because tests may need to reproduce them. + +### 4.1 `domId` stability filter + +A `domId` is recorded only if it passes all of: + +- length between 2 and 64 +- contains at least one lowercase letter +- does not match a UUID-like pattern (`/^[0-9a-f-]{12,}$/i`) +- does not match a hash-like pattern (`/^[a-zA-Z0-9_-]{6}$/` with high + entropy by Shannon estimate ≥ 4.5 bits/char) +- is unique on the page at capture time + +If filtered, `domId` is `undefined` and `hints.domIdLooksStable = false`. + +### 4.2 Class filter + +A class survives into `stableClasses` only if: + +- it is not in the configured CSS-in-JS pattern list (default: + `/^css-[a-z0-9]{4,}$/`, `/^_[a-zA-Z0-9_]{6,}$/`, + Material/Emotion-style hash patterns) +- it is not transient: not in `is-active`, `is-hover`, `is-focused`, + `is-selected`, etc. (configurable list) +- it appears on at most 100 elements at capture time (uniqueness signal; + configurable threshold) + +The full filtered set is preserved in `hints.classesAreCssInJs` so we +know whether class-based matching is even worth attempting. + +### 4.3 Text normalization + +- Trim and collapse whitespace. +- Truncate to 200 characters; original length stored separately. +- Detect dynamic patterns (interpolation markers, dates, counts, + user-specific strings) and set `hints.textIsLikelyDynamic`. The + detector is heuristic — see `prompts/examples/` for cases. + +### 4.4 Parent chain + +We store **at most 3 levels up**, stopping at: + +- the test view root (Datagrok view boundary) +- a `` or document root +- a parent we can already identify uniquely by `testId` or `dgWidgetType` + +Storing more is wasteful; storing fewer makes structural matching too +weak in deeply nested viewers. + +## 5. Visual layer (feature-flagged) + +By default the visual layer is **off**. Reasons: + +- Doubles capture time (screenshot + hash) on every green run. +- Increases registry size by 1–10 KB per element when screenshots are + retained, plus the visual hash itself. +- Per-deployment privacy considerations may apply to screenshot content. + +When the `screenshots` feature flag is on: + +- `visual.boundingBox` and `visual.viewportSize` are always captured + (cheap, no image content). +- `visual.visualHash` is a perceptual hash (default: pHash) of the + element-only screenshot, computed locally. No image data leaves the + test machine if screenshots themselves are not retained. +- `visual.screenshotRef` is set only when image retention is enabled, + pointing to a file under `registry/screenshots/.webp`. + +The offline prompt **may** include a screenshot as an `image` content +block when the flag is on; the prompt template handles the absence +gracefully. + +## 6. Storage + +The registry is a **versioned directory of JSON files**. Recommended layout: + +``` +libraries/self-healing-locators/registry/ + v1/ + packages/ + / + .fingerprints.json # all anchors for one test file + e2e/ + / + .fingerprints.json + screenshots/ # only if flag on + .webp + config/ + self-healing.config.json + css-in-js-patterns.json +``` + +### 6.1 Why a central registry, not co-located files + +We considered three options: + +- **Option A — co-located** (`my.spec.ts` + `my.fingerprints.json` next to it). + Pros: discoverability, atomic moves with tests. + Cons: every test directory grows; harder to scan all fingerprints across + the repo for migrations or audits; spreads test concerns into product + package directories. + +- **Option B — central registry** (this proposal). + Pros: clear ownership boundary; bulk operations possible; schema + migrations live in one place; offline tools have a single source. + Cons: rename-a-test requires updating two places; PRs that touch many + tests touch the registry. + +- **Option C — Datagrok entity** (store in the platform itself). + Pros: collaboration features, sharing, history. + Cons: tests need to talk to the platform to read fingerprints, breaking + P5 (offline-capable). Also creates a circular dependency: platform tests + depend on the platform being up. + +We choose **B**. The registry is a sibling under +`libraries/self-healing-locators/registry/` so the library and its data +ship together. This is one of the open questions in the PR description — +reviewers may push for co-location, in which case we adapt. + +### 6.2 File granularity + +One JSON file per **test file**, not per test or per anchor. This balances: + +- not too coarse (one giant file per package → merge conflicts) +- not too fine (one file per anchor → thousands of tiny files) + +Within the file, anchors are keyed by `anchorName`. The capture API +guarantees stable names; see `api/healing-api.md`. + +### 6.3 Versioning + +Schema version is in every file. The library's loader checks the version +on read. Mismatch behavior: + +- Older version, known migration → migrate in memory, log a warning, write + back on next capture. +- Older version, no migration → fail closed. Manual intervention required. +- Newer version than library knows → fail closed. Library is too old. + +## 7. Capture lifecycle + +Two capture modes coexist; both produce identically-shaped fingerprints. + +### 7.1 Passive (auto-capture on green runs) + +When a test calls `healing.resolve(selector)` and the primary selector +**works** (no fallback needed), the resolver records or refreshes the +fingerprint for that anchor. Cost: a few milliseconds per call. + +Auto-derived `anchorName`: `::`. Stable across runs +of the same test, brittle if the test author changes the selector string +(this is fine — that's a deliberate change). + +### 7.2 Explicit (`healing.anchor()`) + +```ts +const submit = await healing.anchor('login-submit', page.locator('#submit')); +await submit.click(); +``` + +The author commits to a name. Renaming is a deliberate refactor with a +deprecation path (see `api/healing-api.md`). + +Use explicit anchors for: + +- Critical-path elements where heal decisions need extra scrutiny. +- Elements whose default auto-derived name would be ambiguous (selectors + built dynamically). +- Elements where the author wants to attach metadata (`role`, `purpose`) + that informs the offline prompt. + +### 7.3 What we never do at capture + +- Capture during a flaky run. The library refuses to update a fingerprint + if the runtime resolver had to fall back to any tier beyond Tier 0 + (primary selector worked exactly). +- Capture sensitive content. The text normalizer has a configurable + redaction list; values matching it are stored as `` and the + fingerprint records `text.valueAtCapture` only if the field is + whitelisted (e.g. button labels, never password fields). + +## 8. Lifecycle: when fingerprints are deleted + +- When a test is deleted (CI hook prunes orphaned anchors, opens + cleanup PR). +- When `anchorName` changes (old entry kept for one minor version with + `deprecated: true` flag, then pruned). +- Manually, via `grok-heal registry prune` for stale or known-broken + entries. + +The registry is not write-only. It must be maintainable. + +## 9. Open questions for review + +1. **Storage location** — central under the library, or co-located with + tests? See § 6.1. +2. **Per-anchor TTL** — should fingerprints captured before a known + platform breaking-change release auto-expire? +3. **Sensitive value handling** — is the redaction list approach enough, + or do we need a per-test opt-in for `valueAtCapture`? +4. **Screenshot retention** — if the flag is on, do we store images in + the registry (large, slow) or only ephemerally during one CI run? diff --git a/libraries/self-healing-locators/docs/03-confidence-model.md b/libraries/self-healing-locators/docs/03-confidence-model.md new file mode 100644 index 0000000000..6af3429714 --- /dev/null +++ b/libraries/self-healing-locators/docs/03-confidence-model.md @@ -0,0 +1,204 @@ +# 03 — Confidence Model + +> Status: full detail. The confidence formula gates every healing decision. +> Weights are calibratable; the formula's shape is not. + +## 1. What the confidence score is — and isn't + +It is: a number in `[0, 1]` indicating how strongly the candidate element +matches the recorded fingerprint, computed from objective feature +overlaps. + +It is not: an estimate from a language model. Claude does not produce a +confidence number that we use directly. When the offline tool processes +LLM proposals, the same scorer that runs at runtime re-scores each +proposal against the live page. This is intentional — it puts the LLM +and the runtime in the same evaluative frame and prevents the LLM from +"talking itself" into a heal. + +## 2. Formula + +For a candidate element matched against a stored fingerprint: + +``` +score = sum_over_features( w_i * m_i ) / sum_over_features( w_i ) + +where: + w_i = configured weight for feature i + m_i = match value in [0, 1] for feature i (0 if feature is absent + in either the candidate or the fingerprint) +``` + +This is a **weighted Jaccard-style ratio**, not an unbounded sum. +Properties we get for free: + +- Score stays in `[0, 1]`. No artificial caps or normalizations later. +- Missing features on either side don't penalize disproportionately — + they just don't contribute weight. +- Adding new features changes the denominator too, so an existing + fingerprint with a missing new feature isn't suddenly downscored. + +## 3. Default weights + +Default weights, by tier. These are **starting values**. The eval suite +will calibrate them on real Datagrok cases before the system is trusted +in production. + +### Tier 1 — semantic + +| Feature | Weight | Match function | +|---|---|---| +| `testId` exact | 1.00 | binary | +| `domId` exact (passes stability filter) | 0.90 | binary | +| `ariaLabel` exact | 0.85 | binary | +| `ariaRole` + accessible `name` exact | 0.85 | binary | +| `dgWidgetType` + `dgViewerType` exact | 0.80 | binary | +| `ariaLabel` fuzzy (Levenshtein ≥ 0.85) | 0.55 | proportional to similarity | + +### Tier 2 — structural + +| Feature | Weight | Match function | +|---|---|---| +| `tagName` exact | 0.20 | binary | +| `stableClasses` Jaccard | 0.40 | size-of-intersection / size-of-union | +| `parentChain` match (any depth) | 0.35 | per-link partial credit | +| `indexInParent` match | 0.10 | binary, last resort | + +### Tier 3 — text + +| Feature | Weight | Match function | +|---|---|---| +| `visibleText` exact (after normalization) | 0.70 | binary | +| `visibleText` fuzzy (Levenshtein ≥ 0.85) | 0.45 | proportional | +| `placeholder` exact | 0.40 | binary | +| `title` exact | 0.30 | binary | + +When `hints.textIsLikelyDynamic` is true, these weights are halved. We +do not trust dynamic text as a strong identifier. + +### Tier 4 — visual / positional + +| Feature | Weight | Match function | +|---|---|---| +| `boundingBox` IoU | 0.25 | Intersection-over-Union ratio | +| `visualHash` similarity | 0.20 | Hamming distance to similarity, threshold ≥ 0.9 | + +When the screenshot feature flag is off, the visual layer contributes +nothing. The denominator shrinks accordingly. + +## 4. Thresholds + +The decision gate has three bands: + +| Band | Range | Action | +|---|---|---| +| HIGH | `score ≥ 0.80` | Heal silently. Audit log, no flag. | +| MID | `0.55 ≤ score < 0.80` | Heal AND flag for human review. Queue offline. | +| LOW | `score < 0.55` | Fail the test. Queue offline. | + +Additional gates that override the band (these are hard rules, not score +adjustments): + +- **Ambiguity rule.** If two candidates score within 0.10 of each other + in HIGH or MID, demote both to LOW. We never silently pick between + near-tied candidates. +- **Semantic mismatch.** If `ariaRole` of the candidate differs from the + fingerprint's `ariaRole` (e.g. fingerprint says `button`, candidate is + `link`), demote the candidate by one band. This catches "found + something that looks similar but is the wrong kind of thing." +- **Action mismatch.** If the test action is `click` and the candidate + is not interactive (no `tabindex`, no `onclick`, not a focusable role), + demote by one band. +- **Datagrok widget mismatch.** If `dgWidgetType` is recorded and the + candidate's widget type differs, demote by one band. The platform's + semantic layer is too informative to ignore. + +## 5. Why these defaults + +A few non-obvious choices: + +**Tier 1 dominates.** If `testId` matches, the score is ≥ 1.00 / (sum of +present features), which alone is enough for HIGH on a typical +fingerprint. This is the right outcome — when a stable semantic anchor +matches, we should be highly confident regardless of structural drift. + +**Structural is the smallest tier in aggregate.** Sum of structural +weights ≈ 1.05. Structure is a tiebreaker, not a primary signal. SPAs +restructure constantly. + +**Text is heavier than structure but conditional.** Strong when stable; +halved when dynamic. The conditional logic prevents +`visibleText='Welcome, Alice'` from anchoring a test that runs as Bob. + +**Visual is the lightest.** Visual signals are noisy (rendering, +fonts, themes, viewport sizes vary). They're a sanity check, not a +foundation. + +## 6. Calibration + +Defaults will be calibrated against the eval fixtures (see +[`09-eval-and-rollout.md`](./09-eval-and-rollout.md)). The calibration +target: maximize true heals at HIGH while keeping false heals at HIGH +below a configured budget (proposal: < 1%). + +The eval suite reports per-tier contribution, so we can see whether, +say, `dgWidgetType` is overweighted in practice or whether +`stableClasses` is doing nothing useful and can be deprioritized. + +Weights live in `self-healing.config.json`; calibration changes ship as +config commits, not code commits. + +## 7. Why a formula and not a model + +We considered training a small classifier or learned reranker. Rejected +for now because: + +- We don't have labeled data yet. +- Explainability matters: a reviewer asking "why did this heal?" gets a + per-feature breakdown from the formula. A learned model would need + separate explainability tooling. +- The formula has fewer than 15 numbers to tune. A model would have many + more, with attendant overfitting risk on a small corpus. +- If the eval suite shows the formula is fundamentally inadequate, we + swap the scorer behind the same `Scorer` interface. The architecture + permits this. + +## 8. How the offline tool uses the same scorer + +The offline CLI processes a queued case as follows (full details in +[`05-offline-flow.md`](./05-offline-flow.md)): + +1. Load the fingerprint and the live page (via headless browser). +2. Ask Claude for up to 5 candidate selectors. +3. For each candidate, find the element on the page. +4. **Re-score the candidate using the runtime scorer** against the + fingerprint. +5. Rank candidates by their runtime score, not by Claude's preference + order. +6. Apply the same band gates as runtime. + +This means a "great" Claude proposal that scores LOW under our scorer +gets rejected. A "weak-looking" Claude proposal that, on inspection, +recovers Tier 1 features the resolver missed gets accepted. The scorer +is the source of truth. + +## 9. Audit log fields related to scoring + +Every heal event includes: + +```json +{ + "score": 0.83, + "band": "HIGH", + "matched_features": [ + {"name": "ariaLabel", "weight": 0.85, "match": 1.0}, + {"name": "ariaRole+name", "weight": 0.85, "match": 1.0}, + {"name": "stableClasses", "weight": 0.40, "match": 0.66} + ], + "absent_features": ["testId", "domId", "dgWidgetType"], + "demotions_applied": [], + "ambiguity_check": "passed" +} +``` + +Reviewers can reconstruct the decision exactly. No black box. diff --git a/libraries/self-healing-locators/docs/04-runtime-flow.md b/libraries/self-healing-locators/docs/04-runtime-flow.md new file mode 100644 index 0000000000..1059b7695e --- /dev/null +++ b/libraries/self-healing-locators/docs/04-runtime-flow.md @@ -0,0 +1,251 @@ +# 04 — Runtime Flow + +> Status: full detail. This describes what happens during a single test +> run, step by step, with timing and failure modes. + +## 1. Entry point + +A test calls `healing.resolve(selector, options?)` instead of, or as a +wrapper around, `page.locator(selector)`. The resolve function is the +only public entry point that triggers the resolver chain. + +```ts +// Drop-in for Playwright +const submit = await healing.resolve(page, '[data-testid="submit"]', { + anchorName: 'login-submit', // optional; auto-derived if absent + action: 'click', // hint for the action-mismatch gate +}); +await submit.click(); +``` + +The `healing.anchor()` helper (see `api/healing-api.md`) is a thinner +wrapper that always enforces an explicit name and is preferred for +critical-path elements. + +## 2. Step-by-step flow + +``` +┌───────────────────────────────────────────────────────────────────┐ +│ 0. Try the primary selector with a short timeout (default 1s). │ +│ │ +│ Hit → record a passive fingerprint refresh, return element. │ +│ Miss → enter the resolver. │ +└───────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌───────────────────────────────────────────────────────────────────┐ +│ 1. Load fingerprint for `anchorName` from the registry. │ +│ │ +│ Found → continue. │ +│ Not found → fail loudly. Self-healing requires a fingerprint; │ +│ we never invent one from a missing selector. │ +└───────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌───────────────────────────────────────────────────────────────────┐ +│ 2. Run Tier 1 (semantic). │ +│ │ +│ Build candidate selectors from the fingerprint: │ +│ - [data-testid=""] (if recorded) │ +│ - # (if hint says stable) │ +│ - [role=""][aria-label="