test(runner): add isolated repo-readiness e2e framework#1384
Draft
Michael Price (michael-pr) wants to merge 2 commits into
Draft
test(runner): add isolated repo-readiness e2e framework#1384Michael Price (michael-pr) wants to merge 2 commits into
Michael Price (michael-pr) wants to merge 2 commits into
Conversation
WalkthroughAdds a repo-readiness E2E harness that materializes repo shapes into temporary projects, runs the built CLI through node and binary channels, checks JUnit output and project pollution, and reports pass/fail results. It also adds fixture flows and shape data, suite registration and a runner entrypoint, a Bun script, and supporting documentation plus toolchain exclusions for the new E2E files. Sequence Diagram(s)sequenceDiagram
participant RunTs as e2e/run.ts
participant SuitesIndex as getSuite()/allSuites()
participant Channels as resolveChannels()
participant RunCaseFn as runCase()
participant SpawnCli as spawnCli()
participant Reporter as report()
RunTs->>SuitesIndex: resolve suite name or list suites
RunTs->>Channels: build dist/cli.js and dist/qawolf
RunTs->>RunCaseFn: run RepoShape cases for selected channels
RunCaseFn->>SpawnCli: run flows run in a temp project
RunCaseFn->>RunCaseFn: parse JUnit, scan pollution, collect assertions
RunTs->>Reporter: render JSON or TTY results
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Comment |
Contributor
Author
|
CodeRabbit (@coderabbitai) review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.oxlintrc.json:
- Around line 111-117: Keep import-boundary linting enabled for the e2e/**/*.ts
override in .oxlintrc.json and replace the blanket eslint/no-restricted-imports
off setting with a narrower exception for only the specific imports the e2e
harness truly needs. Update the override so the restriction still protects e2e
code from internal or Bun-only imports while allowing only the approved
patterns, preserving the intent of the existing lint boundary rules.
In `@e2e/fixtures/flows/simpleNav.flow.ts`:
- Around line 7-8: The smoke flow in test("navigates") currently depends on
external network access via page.goto("https://example.com"), which makes the
suite brittle. Replace that navigation with an offline-safe approach such as
page.setContent() or a data: URL so the flow only exercises local browser
readiness. Keep the change within the navigates test in simpleNav.flow.ts and
avoid any egress-dependent URL.
In `@e2e/harness/materialize.ts`:
- Around line 26-29: The writeProjectFile helper currently trusts ShapeFile.path
and can write outside the temp project if the path is absolute or contains ../
segments. Update writeProjectFile to resolve the target path against projectDir
and validate that it stays within the project root before
mkdirSync/writeFileSync are called; reject any path that escapes that root using
the existing join/dirname/absPath flow as the place to enforce the check.
In `@e2e/harness/spawnCli.ts`:
- Around line 24-43: The spawnCli harness can hang forever because spawn() is
created without any timeout or kill behavior. Update the spawnCli promise to
enforce a bounded timeout on the child process, terminate it with a kill signal
when exceeded, and resolve with exitCode -1 while preserving any captured
stdout/stderr. Make sure the timeout path is handled alongside the existing
child.on("error") and child.on("close") logic so stalled CLI or browser
downloads fail fast with a clear diagnostic.
In `@e2e/README.md`:
- Line 142: The ordered-list item in the README uses a non-preferred prefix and
will trigger MD029; update the list entry under the suite registration
instructions to use the repo’s standard ordered-list style with 1. so the
markdown linter treats it correctly. Locate the affected text in the README
section that mentions registering the suite in suites/index.ts and adjust only
the list marker format.
- Around line 158-180: The fenced repository layout block in README.md is
missing a language tag and should be updated to satisfy the markdown linter. Add
the appropriate fence language to that block so it is explicitly marked as text,
keeping the documentation lint-clean.
In `@e2e/run.ts`:
- Around line 49-62: `runSuite` is silently treating an unsupported
single-channel selection as success after still doing `build:binary`. Update
`selectChannelNames`/`runSuite` so that when `args.channel` resolves to no
supported channels, the run fails fast or emits a clear warning before the build
step. Use the `selectChannelNames`, `runSuite`, and `resolveChannels` flow to
detect the empty selection and prevent `report` from returning a false green.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 388b1cab-9d55-4c46-a385-9f081aa51c19
📒 Files selected for processing (21)
.oxfmtrc.json.oxlintrc.jsone2e/README.mde2e/fixtures/flows/nativeAndVersioned.flow.tse2e/fixtures/flows/simpleNav.flow.tse2e/fixtures/shapeFiles.tse2e/fixtures/shapes.tse2e/harness/assertions.tse2e/harness/channels.tse2e/harness/materialize.tse2e/harness/report.tse2e/harness/runCase.tse2e/harness/spawnCli.tse2e/harness/tmpWorkspace.tse2e/harness/types.tse2e/run.tse2e/suites/index.tse2e/suites/repoReadiness.tsknip.config.tspackage.jsontsconfig.json
Contributor
Author
|
CodeRabbit (@coderabbitai) review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.oxlintrc.json:
- Around line 115-129: The import restriction in the ESLint config only blocks
the `~` alias, so `e2e` tests can still reach `src/` through relative paths.
Update the `eslint/no-restricted-imports` rule in `.oxlintrc.json` to also
forbid relative imports that resolve into `src/`, keeping the black-box boundary
enforced for nested `e2e/**/*.ts` files.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 691a3b3a-bead-45ac-9bcf-e966d9630f37
📒 Files selected for processing (6)
.oxlintrc.jsone2e/README.mde2e/harness/materialize.tse2e/harness/runCase.tse2e/harness/spawnCli.tse2e/run.ts
Comment on lines
+115
to
+129
| "eslint/no-restricted-imports": [ | ||
| "error", | ||
| { | ||
| "patterns": [ | ||
| { | ||
| "group": ["~/*", "~/**"], | ||
| "message": "e2e/ must not import from src/ — it treats the CLI as a subprocess black box." | ||
| }, | ||
| { | ||
| "group": ["bun", "bun:*"], | ||
| "message": "e2e/ harness uses node: builtins for portability — avoid Bun module imports." | ||
| } | ||
| ] | ||
| } | ||
| ] |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python - <<'PY'
import json, pathlib
cfg = json.loads(pathlib.Path(".oxlintrc.json").read_text())
for override in cfg["overrides"]:
if override.get("files") == ["e2e/**/*.ts"]:
rule = override["rules"]["eslint/no-restricted-imports"]
patterns = rule[1]["patterns"]
groups = [g for p in patterns for g in p.get("group", [])]
print("e2e restricted-import groups:")
for g in groups:
print(" -", g)
print("\nBlocks alias imports into src/:", any(g.startswith("~") for g in groups))
print("Blocks relative imports into src/:", any(g.startswith("..") and "src" in g for g in groups))
break
PY
rg -nP --glob 'e2e/**/*.ts' 'from\s+["'\''](\.\./)+src/' e2e || trueRepository: qawolf/cli
Length of output: 288
🏁 Script executed:
#!/bin/bash
set -euo pipefail
wc -l .oxlintrc.json
sed -n '100,145p' .oxlintrc.jsonRepository: qawolf/cli
Length of output: 1451
🏁 Script executed:
#!/bin/bash
set -euo pipefail
nl -ba .oxlintrc.json | sed -n '105,135p'Repository: qawolf/cli
Length of output: 188
Close the relative-import path into src/. .oxlintrc.json:115-129 only blocks the ~ alias, so nested e2e/**/*.ts files can still reach src/ via relative imports. Add a restricted-import pattern for relative paths into src/ to keep the black-box boundary enforced.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.oxlintrc.json around lines 115 - 129, The import restriction in the ESLint
config only blocks the `~` alias, so `e2e` tests can still reach `src/` through
relative paths. Update the `eslint/no-restricted-imports` rule in
`.oxlintrc.json` to also forbid relative imports that resolve into `src/`,
keeping the black-box boundary enforced for nested `e2e/**/*.ts` files.
Source: Path instructions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relates to WIZ-10907
Overview of Changes
Adds a standalone, manually-run E2E framework under
e2e/that drives the real built CLI as a subprocess black box across 8 repo/monorepo shapes on both the Node and compiled-binary channels. It codifies the ad-hoc matrix used to verify the WIZ-10907 runtime-deps fix, replacing it with a reproducible, data-driven gate. The framework is not bundled intodist/cli.jsordist/qawolfand has no effect on the shipped product.Design highlights:
bun run build:binary), then spawnsnode dist/cli.js …anddist/qawolf …directly. No global install, nonpm link, no~/.local/binmutation.QAWOLF_RUNTIME_DIRat a throwaway dir (warmed once per channel) and materializes each shape into a fresh tmp project dir, so it never touches~/Library/Application Support/qawolf-nodejs. Imports nothing fromsrc/.RepoShapetofixtures/shapes.ts; add a suite = one file insuites/+ one registry line.Testing
Run the full suite manually (builds both artifacts once, then runs 16 cells — 8 shapes × node + binary):
Expected: all 16 cells pass (exit 0, JUnit failures=0, ≥1 test each, zero
node_modulespollution in project dirs), and the global runtime dir is untouched. Verified locally: 16/16 green, zero pollution, global runtime untouched, full local gate green (typecheck/lint/knip/format, 1075 tests).Single channel for faster iteration:
Inspect tmp project dirs after a run (skips cleanup):
Case-08 (
08-native-and-versioned-deps) confirmssharp(native module) anddiff@^8.0.3resolve withoutFILE_HEADERS_ONLY,Could not load the "sharp" module, orCannot find packageon both channels — verifying the inner-hop version-shadowing fix holds on the compiled binary.Checklist