[Design] Self-Healing Locators for Datagrok UI Tests

PR_DESCRIPTION.md

@@ -0,0 +1,124 @@
+# [Design] Self-Healing Locators for Datagrok UI Tests
+
+> **Status:** Design proposal. No executable code in this PR.
+> **Goal:** Get reviewer alignment on architecture, fingerprint schema, prompt
+> contract, healing policy, and rollout plan **before** any code is written.
+
+## TL;DR
+
+UI tests in Datagrok (package tests on Puppeteer, plus a growing Playwright E2E
+suite) break when the DOM, CSS, or labels shift. This proposal introduces a
+two-phase **self-healing locator** system:
+
+- **Runtime fallback** (library, no LLM): when a selector misses, a
+  deterministic resolver walks a 4-tier priority hierarchy (semantic IDs →
+  stable structural → text content → visual/positional) using a
+  **fingerprint** captured during green test runs. A confidence score gates
+  the outcome: high → heal silently, mid → heal + flag for review, low →
+  fail the test.
+- **Offline maintenance** (CLI, uses Claude API): low-confidence and
+  unresolved cases land in a queue. A separate tool prompts Claude with the
+  fingerprint + accessibility tree, validates each proposed selector against
+  a live page in headless Puppeteer/Playwright, and opens a follow-up PR
+  with codemod'd test source updates.
+
+The two phases share one fingerprint schema, one confidence model, and one
+audit log format. The LLM is the **last resort**, not the first move.
+
+## What's in this PR (design only)
+
+```
+libraries/self-healing-locators/
+  README.md
+  docs/
+    01-design.md                 [FULL]
+    02-fingerprint-spec.md       [FULL]
+    03-confidence-model.md       [FULL]
+    04-runtime-flow.md           [FULL]
+    05-offline-flow.md           [FULL]
+    06-datagrok-integration.md   [COMPACT]
+    07-policy.md                 [FULL]
+    08-review-process.md         [COMPACT]
+    09-eval-and-rollout.md       [COMPACT]
+  prompts/
+    system.md                    [FULL]
+    user-template.md             [FULL]
+    examples/
+      01-testid-rename.md
+      02-aria-label-change.md
+      03-structural-refactor.md
+  schemas/
+    fingerprint.schema.json
+    healing-event.schema.json
+    tool-response.schema.json
+  api/
+    healing-api.md               [FULL]
+
+tools/self-healing-cli/
+  README.md
+  docs/
+    cli-overview.md              [COMPACT]
+```
+
+## Decisions baked into this proposal
+
+| Decision | Choice | Where it's discussed |
+|---|---|---|
+| Project location | `libraries/self-healing-locators` + `tools/self-healing-cli` | `01-design.md` |
+| Healing mode | Hybrid: runtime fallback + offline source updates | `04-runtime-flow.md`, `05-offline-flow.md` |
+| LLM provider | Anthropic Claude API | `05-offline-flow.md` |
+| Model strategy | Tiered: Haiku → Sonnet → Opus | `05-offline-flow.md` § "Model selection" |
+| Fingerprint storage | Centralized JSON registry under `libraries/self-healing-locators/registry/` (not in this PR) | `02-fingerprint-spec.md` § "Storage" |
+| Fingerprint capture | Passive (auto-update on green runs) + explicit (`healing.anchor()` API) | `api/healing-api.md` |
+| Screenshots | Feature-flagged, off by default | `02-fingerprint-spec.md` § "Visual layer" |
+| LLM in runtime | **No.** Runtime is deterministic and fast. | `04-runtime-flow.md` |
+| Source code mutation | Offline only, via codemod, gated by PR review | `05-offline-flow.md` § "Codemod & PR" |
+
+## Review focus
+
+Please read in this order — each doc is short and self-contained:
+
+1. **`01-design.md`** — confirm overall shape and component boundaries
+2. **`07-policy.md`** — confirm the "what we heal vs what we let fail" boundary
+3. **`02-fingerprint-spec.md`** — confirm the schema before we lock it in
+4. **`prompts/system.md` + `prompts/user-template.md`** — confirm the contract with Claude
+5. **`03-confidence-model.md`** — confirm the weights and thresholds (these are calibratable, not hardcoded forever)
+6. The rest — orientation only
+
+## Explicit non-goals
+
+- We do **not** propose changes to the Datagrok platform itself in this PR.
+  Any `getWidgetStatus()` extensions or new `data-testid` conventions are
+  noted as recommendations in `06-datagrok-integration.md`, to be addressed
+  separately.
+- We do **not** heal anything beyond locators. Assertions, timing, navigation,
+  and test logic are out of scope. See `07-policy.md` for the rationale.
+- We do **not** mutate test source files automatically without a PR.
+
+## Open questions (please flag in review)
+
+1. **Registry location** — should fingerprints live next to tests, in a
+   sibling directory, or in a single central registry? See
+   `02-fingerprint-spec.md` § "Storage" for trade-offs.
+2. **Failure budget** — what's the acceptable number of LLM-suggested heals
+   per week before we treat it as a signal that the platform needs more
+   stable test IDs?
+3. **PR ownership** — when the offline tool opens a PR, who is the default
+   reviewer? Test author? QA team? Code owners of the affected package?
+4. **Cost ceiling** — concrete dollar limit per offline run before the tool
+   pauses and asks for human approval.
+
+## After this PR is approved
+
+- Iterate on review comments until docs are signed off.
+- Open implementation PRs in this order:
+  1. `libraries/self-healing-locators` core types + fingerprint capture
+  2. Runtime resolver + confidence scorer (no LLM)
+  3. Audit log + healing queue writer
+  4. `tools/self-healing-cli` skeleton + Claude client
+  5. Prompt + tool_use response validation
+  6. Candidate validator (headless browser)
+  7. Codemod + PR opener
+  8. Eval harness + first 30-50 fixtures
+
+Each step is independently reviewable and shippable behind a feature flag.

libraries/self-healing-locators/README.md

@@ -0,0 +1,52 @@
+# @datagrok-libraries/self-healing-locators
+
+Resilient UI element discovery for Datagrok tests. When a primary selector
+breaks because of a UI change, this library finds the same element through
+alternative signals and reports its confidence in the match.
+
+## Status
+
+**Design phase.** This directory currently contains design docs, schemas, and
+prompt specs only. No runtime code has been written yet. See
+[`docs/01-design.md`](./docs/01-design.md) for the proposed architecture and
+[`PR_DESCRIPTION.md`](../../PR_DESCRIPTION.md) at the repo root for the rollout
+plan.
+
+## What this library does (when implemented)
+
+- Captures **fingerprints** of UI elements during green test runs — semantic
+  IDs, ARIA roles, stable classes, parent chain, visible text, optional
+  visual hash, and Datagrok widget metadata via `getWidgetStatus()`.
+- At runtime, when a primary selector misses, walks a 4-tier priority
+  hierarchy (semantic → structural → text → visual) to relocate the
+  element, **without calling an LLM**.
+- Computes a **confidence score** for the match. High confidence heals
+  silently; mid-confidence heals but flags for review; low confidence fails
+  the test and queues the case for offline analysis.
+- Writes an **audit log** for every healing decision.
+
+The companion CLI in [`tools/self-healing-cli`](../../tools/self-healing-cli)
+handles offline maintenance: it consumes the queue, calls Claude to propose
+new selectors for hard cases, validates them against a live page, and opens
+a PR with updated test sources.
+
+## What this library does **not** do
+
+- Does not call any LLM at runtime. Tests stay fast and offline-capable.
+- Does not heal assertions, timing, or test logic. Locators only.
+- Does not mutate test source files. Source edits happen via codemod in the
+  offline CLI, behind a PR.
+
+## Quick links
+
+- [Architecture overview](./docs/01-design.md)
+- [Fingerprint schema](./docs/02-fingerprint-spec.md)
+- [Confidence model](./docs/03-confidence-model.md)
+- [Runtime flow](./docs/04-runtime-flow.md)
+- [Offline flow](./docs/05-offline-flow.md)
+- [Healing policy](./docs/07-policy.md)
+- [Public API](./api/healing-api.md)
+
+## License
+
+Same as the parent repository.

libraries/self-healing-locators/api/healing-api.md

@@ -0,0 +1,194 @@
+# Healing API
+
+> Status: full detail. This is the surface test authors see and depend on.
+
+## Module
+
+```ts
+import * as healing from '@datagrok-libraries/self-healing-locators';
+```
+
+The library is framework-agnostic at the type level. Concrete adapters
+ship for Puppeteer (used by Datagrok package tests) and Playwright
+(used by the standalone E2E suite). The core API works the same way
+through either adapter.
+
+## Configuration
+
+Done once per test process, typically in a global setup file.
+
+```ts
+healing.configure({
+  registryPath: 'libraries/self-healing-locators/registry/v1',
+  auditLogPath: 'test-output/self-healing/locator-events.jsonl',
+  queuePath:    'test-output/self-healing/healing-queue.jsonl',
+  thresholds: { high: 0.80, mid: 0.55 },
+  enableScreenshots: false,
+  enableTier4Visual: false,
+  primarySelectorTimeoutMs: 1000,
+  perTierBudgetMs: { tier1: 200, tier2: 300, tier3: 200, tier4: 500 },
+  // Optional overrides; see docs/03-confidence-model.md for defaults
+  weights: undefined,
+});
+```
+
+`configure()` is idempotent. Calling it twice with identical options is
+a no-op. Calling it with conflicting options throws — config is global
+state, but it's not mutable global state.
+
+## Core: `resolve()`
+
+The drop-in replacement for `page.locator(selector)`.
+
+```ts
+const element = await healing.resolve(page, '[data-testid="submit"]', {
+  anchorName?: string;          // explicit name; auto-derived otherwise
+  action?: 'click' | 'type' | 'assert' | 'read' | 'hover';
+  purpose?: 'normal' | 'destructive' | 'auth';  // policy hint
+});
+```
+
+Returns the framework's element handle (Puppeteer `ElementHandle`,
+Playwright `Locator`). Callers use it normally — `await element.click()`,
+etc.
+
+Behavior:
+
+1. Tries the primary selector (with the configured timeout).
+2. On miss, runs the resolver chain.
+3. Returns the resolved element on HIGH or MID; throws on LOW or fail.
+4. Writes one audit event per call (success or fail).
+
+If the primary selector hits, `resolve()` is just a passive
+fingerprint-refresh wrapper with negligible overhead.
+
+## Explicit anchors: `anchor()`
+
+Preferred for critical paths. Forces an explicit name and surfaces
+metadata into the fingerprint.
+
+```ts
+const submit = await healing.anchor(page, 'login-submit', {
+  selector: '[data-testid="submit"]',
+  action: 'click',
+  purpose: 'auth',
+});
+await submit.click();
+```
+
+Equivalent to `resolve()` with `anchorName: 'login-submit'`, but:
+
+- The `anchorName` is required, not derived. Renames go through a
+  deprecation path (see § "Renaming anchors").
+- The `purpose` field is recorded in the fingerprint and influences
+  the policy gates (e.g. `auth` requires explicit anchors; the policy
+  refuses to heal auto-derived anchors on auth flows).
+- The library can attach optional fingerprint hints
+  (`hints.elementIsInsideViewer`) without the author having to compute
+  them.
+
+## Renaming anchors
+
+Anchor names are part of the contract between tests and the registry.
+A rename is a deliberate refactor, not a silent fix.
+
+```ts
+healing.deprecateAnchor('old-name', { until: '2026-09-01' });
+const submit = await healing.anchor(page, 'new-name', { ... });
+```
+
+The library:
+
+- Loads both fingerprints; new anchor inherits from old until the
+  deadline.
+- Writes a warning to the audit log on each use of the old name.
+- After the deadline, the old anchor is pruned by `grok-heal registry prune`.
+
+## Reading the audit log
+
+For dashboards, custom reporters, or test-time introspection.
+
+```ts
+const events = await healing.audit.read({ since: '2026-04-01' });
+const counters = healing.audit.snapshot();
+// counters: { total_resolves, primary_hits, tier1_hits, ..., failed }
+```
+
+The audit log is jsonl, append-only. The library exposes a streaming
+reader so consumers do not have to parse it themselves.
+
+## Marking review decisions
+
+After a healing PR is reviewed, decisions feed back to the offline tool
+so it doesn't re-propose the same heal next run.
+
+```ts
+// Programmatic alternative to `grok-heal mark`
+await healing.audit.markCase(caseId, {
+  decision: 'accepted' | 'rejected',
+  reason?: string,
+});
+```
+
+Programmatic marking is mostly for the CLI; humans typically use
+`grok-heal mark` from the command line.
+
+## Error types
+
+```ts
+class HealingError extends Error {
+  caseId: string;
+  reason:
+    | 'no_fingerprint'
+    | 'all_tiers_missed'
+    | 'ambiguous_candidates'
+    | 'low_confidence'
+    | 'platform_version_skew'
+    | 'registry_corrupt'
+    | 'unhealable_destructive'
+    | 'unhealable_auth';
+  details: { /* per-reason fields */ };
+}
+```
+
+`HealingError` extends `Error` so existing test reporters render it
+sensibly. The `caseId` lets a developer pull up the audit entry directly
+from the failure message.
+
+## What the API never does
+
+- **Never returns null silently.** Either resolves to an element or
+  throws. There is no in-band "didn't find it" sentinel.
+- **Never retries on transient failures.** That's the test framework's
+  job. Self-healing fixes locator drift, not flakiness.
+- **Never logs to stdout.** All logging goes through the audit log or
+  a configurable logger callback.
+- **Never makes network calls.** No metrics services, no telemetry, no
+  Anthropic SDK in this package.
+
+## Adapter packages
+
+The core library exposes interfaces; adapters implement them.
+
+```
+@datagrok-libraries/self-healing-locators                  // core
+@datagrok-libraries/self-healing-locators-puppeteer        // adapter
+@datagrok-libraries/self-healing-locators-playwright       // adapter
+```
+
+Adapters are thin: they translate between core types and the
+framework's element/page types, and they implement the framework-specific
+parts of fingerprint capture (taking a screenshot, reading the
+accessibility tree, etc.).
+
+The Datagrok Puppeteer test framework uses the Puppeteer adapter. The
+Playwright E2E suite uses the Playwright adapter. The core API surface
+is identical from the test author's perspective.
+
+## API stability
+
+- `1.0` is the first stable release; signatures above ship as is.
+- Pre-1.0 versions may evolve based on early-adopter feedback from the
+  volunteer package(s) in Stage 2 of the rollout.
+- Breaking changes after 1.0 follow standard semver and require a
+  deprecation window of at least one minor version.