Skip to content

feat: expansion scope — cross-framework heal, MCP server, watch mode, heal history (v0.2.0) - #8

Merged
Desperado merged 2 commits into
mainfrom
feat/expansion-scope
Jul 10, 2026
Merged

feat: expansion scope — cross-framework heal, MCP server, watch mode, heal history (v0.2.0)#8
Desperado merged 2 commits into
mainfrom
feat/expansion-scope

Conversation

@Desperado

Copy link
Copy Markdown
Contributor

Wires the four expansion-scope features (QUA-1271, QUA-1272 + watch/report):

1. Cross-framework heal — Cypress + Selenium (QUA-1272)

New src/ninelives/frameworks/ adapter layer; the run→classify→heal→verify loop is now framework-agnostic.

  • Cypress: npx cypress run --reporter json --quiet in the user's own project (never scaffolds a Cypress install). Locator misses arrive wrapped in AssertionError — new LOCATOR_OVERRIDE_PATTERNS classify them as selector drift so the behavior-vs-drift guard doesn't refuse to heal them.
  • Selenium (Python): runs through the user's own pytest; NoSuchElementException's {"method":…,"selector":…} payload and By.* calls feed selector extraction.
  • Auto-detect per spec (.cy.* / .py / package.json deps), --framework to force. Tier 1 never injects Playwright waitFor into non-Playwright code; Tier 2 prompts carry the framework label + right code fence.

2. 9l mcp — the agent-native wedge (QUA-1271)

Zero-dependency stdio MCP server (newline-delimited JSON-RPC): heal_test + run_test, so Claude Code / Cursor / Codex heal a red test mid-session. claude mcp add 9lives -- 9l mcp. Heal-loop output rerouted to stderr; non-interactive mode saves <spec>.healed instead of prompting; assertion failures come back needs-human with an explicit don't-mask-the-bug note.

3. 9l watch + pre-commit hooks

Poll-based heal-on-save (no OS watcher deps, self-writes don't retrigger) and .pre-commit-hooks.yaml with 9lives-heal / 9lives-run.

4. Heal history + 9l report

Every heal attempt appends to .9lives/history.jsonl next to the spec (local only, best-effort — can never break healing). 9l report [--md] aggregates into a brittle-selector report: heal counts, winning anchors, pin-a-testid recommendations.

Housekeeping

  • __version__ was stuck at 0.1.0 since the first release (9l --version lied on 0.1.1–0.1.3); synced and bumped to 0.2.0 with pyproject.
  • New keywords/classifiers for PyPI discoverability (cypress, selenium, mcp).

Tests: 48 green (20 new — detection, Cypress/Selenium parsing + classification, tier-1 framework guard, history/report, MCP protocol roundtrip, watch scanner). Ruff clean. Not tagged/released yet — v0.2.0 release is a separate step after merge.

… heal history (v0.2.0)

Four expansion features that widen 9lives beyond a Playwright-only CI tool:

* Cross-framework heal (QUA-1272): framework adapter layer under
  src/ninelives/frameworks/. Cypress specs run via 'npx cypress run
  --reporter json --quiet' in the user's own project; Selenium (Python)
  specs run via the user's own pytest. Auto-detected per spec
  (.cy.* / .py / package.json deps), overridable with --framework.
  Cypress locator misses (wrapped in AssertionError) are classified as
  selector drift so the behavior-vs-drift guard doesn't eat them;
  Selenium's NoSuchElementException JSON payload feeds selector
  extraction. Tier 1 never injects Playwright waits into non-Playwright
  code; Tier 2 prompts are framework-labelled.

* MCP server (QUA-1271): '9l mcp' — zero-dependency stdio JSON-RPC
  implementing initialize / tools/list / tools/call with heal_test and
  run_test, so Claude Code / Cursor / Codex heal red tests in-loop.
  Heal-loop output is rerouted to stderr to keep the protocol stream
  clean; non-interactive mode saves a .healed copy instead of prompting.

* 9l watch + pre-commit hooks: poll-based heal-on-save watcher (no OS
  watcher deps) and .pre-commit-hooks.yaml with 9lives-heal / 9lives-run.

* Heal history + 9l report: every heal attempt appends to
  .9lives/history.jsonl (local only); '9l report' aggregates it into a
  brittle-selector report with anchor stats and pin-a-testid
  recommendations, terminal or --md.

Also: sync __version__ (was stuck at 0.1.0 since first release — '9l
--version' lied) and bump to 0.2.0.

48 tests green (20 new), ruff clean.
@sigilix

sigilix Bot commented Jul 10, 2026

Copy link
Copy Markdown

Sigilix Overview

Effort: 4/5 (large)

Quality gates

  • ✅ PR title follows convention
  • ✅ PR description is complete
  • ℹ️ PR is linked to an issue — No Closes #N / Closes SIG-N keyword found in PR body or commit messages.

Summary — latest push

Introduces cross-framework healing (Cypress, Selenium/pytest) via an adapter layer, an MCP server for agent-driven healing, poll-based watch mode, and heal-history tracking with a report command. The heal loop is now framework-agnostic, routing through adapters that handle preflight, execution, and artifact collection, while Tier 1 guards framework-specific injection (e.g., Playwright waits) and classification overrides ensure Cypress/Selenium locator misses are treated as selector drift rather than assertion failures.

Important files

File Score Notes Next step
src/ninelives/cli.py 5/5 Central dispatch for all new commands (watch, report, mcp) and the framework-aware heal/run loops, plus history recording hooks. Extract the argument-parsing boilerplate and command dispatch into a separate module to reduce the 400+ line CLI file size and isolate the command routing logic.
src/ninelives/frameworks/__init__.py 4/5 Framework auto-detection and adapter resolution, including package.json dependency inspection for Cypress vs Playwright disambiguation. Add unit tests for the detection heuristics covering edge cases like a package.json with both cypress and @playwright/test dependencies.
src/ninelives/frameworks/selenium.py 4/5 Selenium/pytest adapter with preflight validation, subprocess execution, and pytest output parsing for NoSuchElementException payloads. Fuzz the pytest output parser with varied traceback formats and multiline error messages to ensure selector extraction doesn't silently fail.
src/ninelives/healing/strategy.py 4/5 Adds LOCATOR_OVERRIDE_PATTERNS for Cypress/Selenium error classification and a framework field to TestFailure to prevent cross-framework Tier 1 misapplication. Verify the override patterns are ordered correctly so they don't accidentally shadow the existing FLOW_CHANGE_PATTERNS for edge-case error messages.
src/ninelives/healing/tier1.py 3/5 Guards timing/scroll injection behind a Playwright-only check and enriches healing metadata with old/new selectors for history tracking. Add a test confirming that a Cypress timing failure skips the Playwright-specific wait injection and escalates to Tier 2.

Sequence diagram

sequenceDiagram
    participant CLI
    participant Adapter as Framework Adapter
    participant Runner as Test Runner
    participant Tier1 as Tier 1 Healer
    participant History as .9lives/history.jsonl
    CLI->>Adapter: get_adapter(spec, framework)
    Adapter->>Adapter: preflight(spec)
    CLI->>Adapter: run(spec)
    Adapter->>Runner: execute spec (pytest/npx/cypress)
    Runner-->>Adapter: RunResult
    CLI->>Adapter: collect(result, dest_root)
    CLI->>Tier1: heal(failure)
    Tier1-->>CLI: HealingResult
    CLI->>History: record_heal(spec, status, ...)
Loading

Confidence: 3/5

The multi-framework adapter pattern and MCP server substantially expand the attack surface with subprocess invocation and agent-callable paths that rely on input validation and path sanitization rather than comprehensive hardening tests.

  • The Selenium adapter's run() method passes the resolved absolute spec path directly to pytest via an argv list — confirm there's no path traversal or flag injection risk if a spec path contains special characters.
  • LOCATOR_OVERRIDE_PATTERNS in strategy.py are checked before FLOW_CHANGE_PATTERNS; verify this ordering doesn't misclassify genuine assertion failures from Cypress/Selenium as locator drift if the error message contains overlapping keywords.
  • The MCP server's serve() entry point is imported lazily in CLI; review the mcp_server module (not in this diff) to ensure it validates tool inputs and doesn't expose arbitrary file reads or command execution.
  • The _record_history function in cli.py silently swallows all exceptions — confirm this won't mask a corrupted .9lives/history.jsonl that should surface as a user-visible warning.
  • The cmd_watch function's on_change callback catches RunnerError but lets other exceptions propagate — verify the watch loop handles unexpected errors without terminating the entire poll cycle.

Suggested labels: breaking-change feature


Posted · 0e537ce · 0 findings — View review
Dismiss @sigilix dismiss <reason> (not-a-bug | bad-anchor | already-covered | too-minor | wrong-context) · Re-run /sigilix review

@sigilix sigilix Bot added the enhancement New feature or request label Jul 10, 2026

@qualitymaxapp qualitymaxapp Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QualityMax Security Scan

Found 2 security issue(s).

Scanned by LLM SAST (gemini-3.1-flash-lite).

Fix all findings with your LLM agent
<your-llm-agent> "Fix all QualityMax security findings in this PR:\n- injection in src/ninelives/frameworks/selenium.py:45: User-controlled spec path is passed to subprocess.run via shell-like command list\n- path-traversal in src/ninelives/mcp_server.py:165: User-provided 'spec' path is expanded and used without validation, allowing access to arbitrary files on the system."

Comment thread src/ninelives/frameworks/selenium.py
Comment thread src/ninelives/mcp_server.py
@sigilix

sigilix Bot commented Jul 10, 2026

Copy link
Copy Markdown

Warning

Sigilix couldn't generate a review for this PR.

Reason: synth_failed_parse — The synth model returned output that couldn't be parsed as JSON, even after a retry with a tightened prompt.

Re-run /sigilix review to try again. Run on ef86a9a.

@Desperado

Copy link
Copy Markdown
Contributor Author

/sigilix review

@sigilix

sigilix Bot commented Jul 10, 2026

Copy link
Copy Markdown

Re-running ensemble review on the current head. Standard sticky comment will update when complete.

- MCP heal_test/run_test: validate the tool-supplied spec path — must be
  an existing file with a recognized test suffix, contained in the
  server's working directory (NINELIVES_MCP_UNRESTRICTED=1 to lift).
  The real risk isn't reading files: healing a spec ultimately EXECUTES
  it (pytest import / test run), so cwd containment is the meaningful
  boundary for a prompt-injected agent.
- Selenium adapter: reject non-.py specs in preflight. The subprocess
  call was already injection-safe (argv list, no shell, resolved
  absolute path can't parse as a pytest flag) — this is defense-in-depth
  for the MCP path.

3 new tests (51 total).
@Desperado

Copy link
Copy Markdown
Contributor Author

Response to QualityMax Security Scan — both findings addressed in 0e537ce

Finding 1 — injection in frameworks/selenium.py (spec path → subprocess.run): partially disputed, hardened anyway.
The claim as stated is a false positive: the call uses an argv list (subprocess.run([pytest_bin, str(spec), ...])) — there is no shell, so no injection surface — and the path is resolve()d to absolute first, so it can't be parsed as a pytest flag either. What was worth taking: only .py files make sense to hand to pytest, and via the MCP surface the path is agent-supplied. preflight() now rejects non-.py specs before anything runs.

Finding 2 — path-traversal in mcp_server.py (unvalidated spec path): accepted, and fixed for a stronger reason than flagged.
The scanner flagged reading arbitrary files, but the sharper risk is that healing a spec ultimately executes it (pytest imports it; Playwright/Cypress run it) — so an agent tricked by prompt injection could get arbitrary code execution by pointing heal_test at any file. New _validated_spec() now enforces: file exists → recognized test-spec suffix → contained in the server's working directory (NINELIVES_MCP_UNRESTRICTED=1 to opt out). Documented in the README and the server's MCP instructions.

Note for the SAST loop: the check-run summary said "3 security finding(s)" but this comment lists 2 — if there's a third below the posting threshold, surfacing it would help.

3 new regression tests; suite at 51 green.

@qualitymaxapp

qualitymaxapp Bot commented Jul 10, 2026

Copy link
Copy Markdown

✅ QualityMax Pipeline

Gate Result
🔍 AI Review ✅ Clean · gemini-3.1-flash-lite
🧪 Repo Tests ❌ 0/0 passed (jest)

Powered by QualityMax — AI-Powered Test Automation

@sigilix sigilix Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📊 Reviewed 13 of 18 changed files across this PR so far — the remaining 5 were below the priority cutoff for this very large PR. Split into smaller PRs to cover them.

@Desperado
Desperado merged commit 4491111 into main Jul 10, 2026
7 checks passed
@Desperado
Desperado deleted the feat/expansion-scope branch July 10, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant