[JeremyYao]

Closes #1462

Description

Adds Playwright end-to-end (E2E) tests. They start a real VS Code and use the extension the way a person would, including inside webviews like the Data Editor. The existing tests can't reach inside webviews, so that UI has only been checked by hand, using the checklist in src/tests/README.md.

The new tests are in their own e2e folder, and the existing tests aren't changed. There are 3 tests so far:

• Opening a file in the Data Editor and checking its bytes
• Changing a byte, saving, and checking the file on disk
• Debugging a schema until it pauses on entry

Why Playwright

#1462 asks for vscode-extension-tester, so this PR is a change of direction. All three tools can test webviews, and vscode-extension-tester can run these tests: the same 3 tests were built with it on jy/extester-e2e-tests to compare them. Playwright was chosen because its tests are simpler to write and failures are easier to diagnose, not because the other tools can't do the job.

	Playwright	vscode-extension-tester 8.28.1
Tests passed, in 8 runs each on VS Code 1.90.0 and 1.144.0	48 of 48	47 of 48
Setup code, without comments	182 lines	106 lines
Tests and page objects, without comments	107 lines	139 lines
Rebuild before a run	56 s	125 s
When an assertion fails	Expected and received values, the line in the test, and a step-by-step trace	A timeout message and a screenshot

These results are from Windows only. The full results and their limits are in the comment below.

• Webview tests are simpler to write. Most of the manual checklist is Data Editor behaviour, like drag selection, tooltips and scrolling, so tests will often move between VS Code's UI and a webview. In the other two tools, a test enters the webview's frames and has to leave them again, using switchToFrame() and switchBack(), or open() and close(), before it can use the rest of VS Code's UI. Playwright's frame locators reach into VS Code's nested iframes directly, so there's nothing to enter or leave.
• Failures are easier to diagnose. Every test keeps a trace with a timeline, screenshots and page snapshots, and CI uploads them when tests fail. That's how the Java bug below was diagnosed on Linux from a Windows machine. vscode-extension-tester keeps a screenshot of each failed test.
• Failures stay in one test. Each test gets a new VS Code. vscode-extension-tester starts one per run, and in the comparison, a dialog stuck open by one test also failed the next two.
• Elements are found fresh each time. A redraw or frame switch can leave a Selenium element reference stale, and Selenium's docs recommend finding the element again each time. Playwright locators do this automatically and wait for elements, so these tests contain no fixed sleeps. The exploratory work in Exploratory: Create simple integration/end-to-end (e2e) tests using vscode-extension-tester #1290 hit a StaleElementReferenceError, and an earlier UI test effort needed delays that varied between machines.
• It fits this repo. vscode-extension-tester's install fails on Node 20, so it needs its own package.json to keep our Node 20 CI jobs installing. It also tests only the latest 3 VS Code versions, and treats 1.90, our minimum, as best-effort. On Windows it has to install a packaged .vsix, because ChromeDriver lowercases --extensionDevelopmentPath and VS Code then ignores it. Playwright installs on Node 20, isn't tied to VS Code releases, and loads the extension directly from the repo.
• WebdriverIO has more moving parts. It would add a second test runner, a third-party service, and a Chromedriver matched to each VS Code version. The earlier discussion also found its docs less thorough, and its example failed with a __dirname error. A workaround was suggested there, but it wasn't tried.
• Others use this approach. VS Code's own smoke tests launch through Playwright's _electron API, and so does Confluent's VS Code extension.

Downsides

• Playwright's Electron support is officially experimental.
• This suite has more code overall: 289 lines against 245. The extra code is setup, which vscode-extension-tester handles more of by default.
• Both other tools include ready-made page objects for VS Code's UI, including webviews. Here, those selectors are hand-written and can break between VS Code releases. They're all in e2e/pages/workbench.ts to limit that.
• vscode-extension-tester has stronger helpers for debug UI.
• The comparison is a small sample from one OS, and WebdriverIO wasn't tried. Its part of this rationale comes from the discussion above and its documentation and source code.

Sources:

• vscode-extension-tester README (https://github.com/redhat-developer/vscode-extension-tester)
• vscode-extension-tester WebviewMixin.ts (https://github.com/redhat-developer/vscode-extension-tester/blob/main/packages/page-objects/src/components/WebviewMixin.ts)
• vscode-extension-tester issue 2450 (WebView.iframe locator matches nothing on VS Code 1.132.0 — switchToFrame() then silently no-ops redhat-developer/vscode-extension-tester#2450)
• vscode-extension-tester issue 2023 (EXTENSION_DEV_PATH appears to have no effect on run-tests redhat-developer/vscode-extension-tester#2023)
• wdio-vscode-service WebView.ts (https://github.com/webdriverio-community/wdio-vscode-service/blob/main/src/pageobjects/workbench/WebView.ts)
• WebdriverIO wdio-vscode-service docs (https://webdriver.io/docs/wdio-vscode-service)
• WebdriverIO VS Code extension testing docs (https://webdriver.io/docs/extension-testing/vscode-extensions)
• Playwright FrameLocator docs (https://playwright.dev/docs/api/class-framelocator)
• Playwright locators docs (https://playwright.dev/docs/locators)
• Selenium errors docs (https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/)
• daffodil-vscode discussion 129 (Automated Testing Framework for GUI Tests ctc-oss/daffodil-vscode#129)

#1462 acceptance criteria

Criterion	Status
VS Code Extension Tester is installed	Replaced by Playwright, for the reasons above
A test instance doesn't bleed into the user's VS Code	Met. Tests use a separately downloaded VS Code, temporary user data and extensions folders, and port 9100 for the Data Editor server.
Behavior is consistent	Mostly met. The tests have no fixed sleeps, but the debugger test relies on the 5-second wait under known issues. In one set of 8 local runs, 3 failed because of it. A later set of 16 runs had no failures.

Bug fix: Daffodil CLI ran on the wrong Java

The debugger test found that a valid schema could fail with "Schema compilation failed. Fix the schema and try again." The real error was an UnsupportedClassVersionError, because the CLI was running on a Java version that was too old. There were two causes:

• Daffodil's launcher scripts ignore JAVA_HOME and run the first java on the PATH.
• Vite replaces process.env with {} when bundling, so on Linux and macOS the CLI ran with no PATH and picked up the system Java.

The CLI now runs with the real environment, with the chosen JDK first on the PATH.

Other changes

• Adds the test:e2e, test:e2e:ui, test:e2e:debug and test:e2e:report scripts
• Adds an e2e CI job on Windows, macOS and Linux, which uploads the HTML report when tests fail
• Adds a testId prop to Button, used by the Data Editor's save buttons
• Includes e2e in yarn lint, and excludes its output from the RAT check
• Removes vscode-extension-tester, which was never used
• Removes @types/glob, which described an older glob API and broke the build once vscode-extension-tester was removed

Known issues not fixed here

The debugger start uses a fixed 5-second wait. After starting the debugger, runScript waits 5 seconds and assumes it's ready. A slow start fails with connect ECONNREFUSED, and a fast start waits longer than it needs to. The wait dates back to at least May 2022 (c24daf3).

From src/utils.ts:

if (type.includes('daffodil')) {
  await delay(5000).then(() => {})
} else {
  const wait_port = require('wait-port')
  await wait_port({ host: '127.0.0.1', port: port, output: 'silent' })
}

The debugger always takes the first branch, because runDebugger passes the type 'daffodil' and no port. Switching it to wait_port is untested, and the port check might look like a client connecting to the debugger.

process.env is empty in the rest of the bundle. getJavaHome can't see JAVA_HOME, and environment variables like OMEGA_EDIT_SERVER_LOG_LEVEL have no effect.

Daffodil jar downloads can be slow or stall. Each debugger test downloads the jars again, because every test gets a new user data folder. The extension sets no timeout on the download. It also only checks that the folder exists, so an interrupted download may leave a partial folder that later runs reuse.

Wiki

• I have determined that no documentation updates are needed for these changes

• I have added the following documentation for these changes

• Added an "End-to-End Tests" section to DEVELOPMENT.md, including how a test run works

Review Instructions including Screenshots

You'll need Node 22 or newer, JDK 17 and sbt. Run yarn install first.

Each test opens its own VS Code window, so windows will open and close during a run. Tests use port 9100 for the Data Editor server, so they won't affect a Data Editor you have open.

To explore the tests:

yarn test:e2e:ui             # interactive viewer: pick tests and inspect each step
yarn test:e2e:debug debugger # step through a test in the Playwright Inspector
yarn test:e2e:report         # report from the last run, with a trace for each test

Confirmation Testing

Already tested

• All 3 tests pass locally on Windows in 8 repeated runs each on VS Code 1.144.0 and 1.90.0 (48 of 48)
• The e2e CI job passes on Windows, macOS and Linux with VS Code stable, and on Linux with VS Code 1.90.0
• A test that fails on purpose leaves a trace in the report
• The interactive viewer was driven with a script: all tests passed, and a passing test showed its actions and page snapshots
• The Inspector opens attached to VS Code. Stepping through a test wasn't tried.

Try it yourself

1. Run the tests on the oldest supported VS Code, which is the default. All 3 should pass.

   yarn test:e2e

2. Run them on the latest VS Code. All 3 should pass again.

   # bash
   DAFFODIL_TEST_VSCODE_VERSION=stable yarn test:e2e
   
   # PowerShell
   $env:DAFFODIL_TEST_VSCODE_VERSION = 'stable'; yarn test:e2e

3. Check that a test can really fail. In e2e/dataEditor.spec.ts, change the expected '54' to '55', then run:

   yarn test:e2e dataEditor
   yarn test:e2e:report

   shows the bytes of the opened file should fail, and its trace should show the first byte as 54. Undo the edit with git checkout -- e2e/dataEditor.spec.ts.

4. Check that the tests stay isolated. Open a Data Editor in your normal VS Code, run yarn test:e2e dataEditor, and check that your editor still works. git status should show no new files afterwards.

The debugger test can fail with connect ECONNREFUSED 127.0.0.1:4711, or stall while downloading the Daffodil jars (see known issues). This happened 4 times in one set of 8 local runs, and not at all in a later set of 16. CI retries failed tests but local runs don't, so re-run it if you see either.

Try the Java fix

This needs Linux, with JDK 17 or newer first on the PATH and an older Java at /usr/bin/java. The bug was found on the ubuntu-22.04 CI runners.

yarn test:e2e debugger                              # passes
git checkout main -- src/daffodilDebugger/utils.ts  # puts the old code back
yarn test:e2e debugger                              # fails
git checkout HEAD -- src/daffodilDebugger/utils.ts  # restores the fix

The second run should fail after about 2 minutes, and its trace should show "Schema compilation failed. Fix the schema and try again."

Regression Testing

Already tested

• No test files under src/tests were changed
• All 43 CI checks pass on this commit, including all 33 Build, Test, and Package jobs
• Those jobs have an existing flake, which hit an earlier run of this branch: the Daffodil Debugger before all hook timed out, then both 4.1.0 tests failed on retry. main failed the same way at 360290c (run 40520441064)
• yarn install --frozen-lockfile passes on Node 20.19.4, so the Node 20 CI jobs still install
• License checks and sbt ratCheck pass

Try it yourself

1. Confirm the existing tests are unchanged. This should print nothing:

   git diff --stat main...HEAD -- src/tests

2. Run the existing sbt, Vitest and Mocha tests. All should pass, apart from the flake above if it occurs.

   yarn test

3. Run the repo checks:

   yarn lint
   sbt ratCheck

   On a Windows checkout with core.autocrlf=true, use yarn prettier src e2e -c --end-of-line auto instead of yarn lint.

4. On Node 20, confirm that the dependency changes still install:

   yarn install --frozen-lockfile

5. Check the production code changes by hand. Press F5 to start the extension, as described under "Debugging the Extension" in DEVELOPMENT.md, then:

   • Data Editor save buttons (Button, FileMetrics): run Daffodil Debug: Data Editor, open a copy of a file, and change a byte. Save… should reveal Save and Save As, and both should work.
   • Schema errors (validateSchemaWithDaffodilCLI): add a deliberate error to a copy of a schema and run Daffodil Debug: Debug File. It should still show "Schema compilation failed. Fix the schema and try again."

Made with assistance from Claude Code

[JeremyYao]

I wanted real evidence for this choice, so I built the same 3 tests with vscode-extension-tester 8.28.1 on jy/extester-e2e-tests. vscode-extension-tester can run these tests, but I still prefer Playwright, even though it takes more code overall.

Both suites test the same extension source. I ran each one 8 times on VS Code 1.90.0 and 8 times on 1.144.0 (stable). These results are all from Windows, so macOS and Linux still need comparing. WebdriverIO wasn't part of this.

	Playwright	vscode-extension-tester
Tests passed	48 of 48	47 of 48
Time for a full run	About 29–34 s	About 40–44 s
Rebuild before a run	56 s	125 s
Setup code, without comments	182 lines	106 lines
Tests and page objects, without comments	107 lines	139 lines
Total code, without comments	289 lines	245 lines
Places that wait explicitly	8	12
Dependencies	3 packages, about 18 MB	348 lockfile entries, 107 MB
VS Code instances	A new one for each test	One shared by all tests
Loading the extension on Windows	Directly from the repo	From a packaged .vsix

Why Playwright, despite the extra code

• The extra code is setup, which is mostly written once. Playwright is 44 lines longer overall, but its tests and page objects are 32 lines shorter, and that's the code that grows with every new test. Much of the setup gives each test its own VS Code and records its trace.
• Failures are easier to diagnose. For the same deliberately broken test, Playwright reported Expected: "55", Received: "54" and the line in the test, and left a 4 MB step-by-step trace. vscode-extension-tester reported Waiting until element text is, with neither value and no line in the test, and left a screenshot.
• Failures stay in one test. With one shared VS Code, a dialog stuck open by one test also failed the next two in an early run. Each Playwright test starts with a new VS Code.
• There are fewer traps when writing tests. Playwright waits for elements automatically. With vscode-extension-tester, I had to work around these:
  • InputBox.create() only waits 5 seconds for the box to become visible, whatever timeout it's given.
  • A WebView object created before its editor opens never recovers, and switchToFrame() then reports that the iframe is missing.
  • driver.wait stops at the first error, so a file being replaced by a save has to be handled by hand.
• It's lighter on the repo. vscode-extension-tester needs Node 22 or newer, so it needs its own package.json to keep our Node 20 CI jobs installing. It also needed a selenium-webdriver version pin to install on Node 24.13.0.
• It can use the build we already have. On Windows, ChromeDriver lowercases --extensionDevelopmentPath, so VS Code ignores it and vscode-extension-tester has to use a packaged .vsix. That's why its rebuild takes more than twice as long. Separately, 8.28.1 clears EXTENSION_DEV_PATH unless coverage is on, which is fixed upstream but not yet released (EXTENSION_DEV_PATH appears to have no effect on run-tests redhat-developer/vscode-extension-tester#2023).

What vscode-extension-tester does better

• It needs less setup, and configures more of VS Code for testing by default.
• It maintains its own selectors for VS Code's UI, while ours are hand-written and can break between releases.
• Testing the packaged .vsix is closer to what users install.

How far to trust these numbers

• One failure in 48 is a small sample, so this doesn't prove that either tool is more reliable.
• vscode-extension-tester's one failure was on 1.90.0, which it only supports on a best-effort basis. It also failed once on 1.90.0 in a warm-up run that I didn't count, and that failure left no screenshot because capturing it timed out.
• Playwright isn't free of flakes either. Its debugger test failed 4 of 8 runs in an earlier local set: 3 from the extension's fixed 5-second wait, and 1 from a stalled Daffodil jar download.

#1462 asks for vscode-extension-tester, so this is still the maintainers' call. If you'd prefer it, the branch above shows that it's workable.