test(e2e): packaged-build smoke suite + Linux x64 release gate (smoke phase 1)

[meinharrd]

Summary

Phase 1 of automating the release smoke test (docs/agent-playbooks/release-process.md §6): launch, version and persistence, run against the artifact a release actually ships instead of against the source tree. Nodes, navigation, the headline feature and upgrade-from-previous-version stay manual (later phases / other platforms).

• test-e2e/fixtures.js — the harness fixture launches through FREEDOM_E2E_EXECUTABLE when that variable is set (executablePath, no '.' argument, plus --no-sandbox when FREEDOM_E2E_NO_SANDBOX=1), and behaves exactly as before when it is not. The scratch profile moved out of electronApp into its own userDataDir fixture so a spec can quit the app and start a second instance on the same on-disk state; that is the new relaunchApp fixture. browserWindow(app) is exported so a relaunched app can be waited on the same way. live-fixtures.js / onboarding-fixtures.js are untouched.
• playwright.config.js — new packaged project over test-e2e/packaged/, with harness narrowed to exclude that directory. A packaged-preflight setup project (dependencies: ['packaged-preflight']) validates FREEDOM_E2E_EXECUTABLE before any Electron process starts: unset, unreadable, not a file, or not executable each fail with the fix in the message and every packaged spec is skipped. (A top-level globalSetup was the first attempt and is wrong here — config.projects in globalSetup is not filtered by --project, so the guard fired on plain npm run test:e2e runs too.)
• test-e2e/packaged/ — launch.spec.js (chrome mounts, exactly one live BrowserWindow, process.versions.electron/chrome non-empty and matching the renderer's UA, app.isPackaged), version.spec.js (app.getVersion() — the value index.js feeds to setAboutPanelOptions, i.e. what About shows — equals FREEDOM_E2E_EXPECTED_VERSION or package.json), persistence.spec.js (theme change survives a full quit and relaunch of the same executable against the same profile).
• npm run test:e2e:packaged.
• .github/workflows/release.yml — new smoke-linux-x64 job: needs: [linux], npm ci --ignore-scripts (nothing is rebuilt here), Playwright system deps via the existing composite action, downloads the freedom-linux-x64 artifact, installs the .deb through scripts/ci/apt-hardening.sh (bounded/retried, like every other apt call on a Linux runner), runs the suite against /opt/Freedom/freedom, then extracts the AppImage and runs it again against squashfs-root/freedom. Reports/traces upload as smoke-linux-x64-report with if: always(). release now needs this job as well, so a Linux x64 artifact that cannot launch, misreports its version, or loses a setting across a restart never reaches a release page. The mac/windows/linux-arm64 jobs and the attach script are unchanged.
• Docs — §6 of the release playbook (what is automated, what still is not, how to run it locally), a bullet in its Appendix B, and the testing section + script table of docs/development.md.

Related issue

Verification

Everything below was run on this branch (b361ab9c), Linux x64, Node 24.19.0.

• npm run lint — clean.
• Relevant unit tests — npm test (see comment below for the summary line; no src/ file changed, so this is a no-regression check).
• Relevant Playwright or live smoke tests:
  • Fixture switch is a no-op when the env var is unset: xvfb-run -a npm run test:e2e -- test-e2e/settings.spec.js → 6 passed (34.0s).
  • Packaged suite against a local build: npm run build -- --linux --x64 (after adblock/ant/ipfs/radicle --linux --x64/myotis downloads; Tor skipped), then FREEDOM_E2E_EXECUTABLE="$PWD/dist/linux-unpacked/freedom" FREEDOM_E2E_NO_SANDBOX=1 xvfb-run -a npm run test:e2e:packaged → 5 passed (27.6s) (4 specs + the preflight setup test).
  • Mutation tests, so the new assertions are known to carry weight:
    • FREEDOM_E2E_EXPECTED_VERSION=0.0.1-bogus → version spec fails (Expected "0.0.1-bogus", Received "0.8.5-dev").
    • relaunchApp temporarily pointed at a fresh profile dir → persistence spec fails (Expected "light", Received "system"); reverted.
    • FREEDOM_E2E_EXECUTABLE pointed at the unpackaged node_modules/electron/dist/electron → both launch specs fail.
    • Preflight: unset → The "packaged" project needs FREEDOM_E2E_EXECUTABLE and it is not set + 4 did not run; /no/such/freedom → "cannot be read"; a non-executable file (package.json) → "is not executable".
• npx prettier --check on every changed file — clean (.github/workflows/release.yml, both docs, package.json, playwright.config.js, test-e2e/fixtures.js, test-e2e/packaged/*.js).
• git diff --check — clean. shellcheck on the new job's run: blocks (extracted from the YAML) — clean. actionlint is not installed on this machine; the workflow parses as YAML and the job graph was asserted programmatically (smoke-linux-x64 needs [linux], release needs [mac-arm64, linux, windows-x64, smoke-linux-x64]).

Runner verification of the workflow change is in a follow-up comment — a workflow_dispatch run on this branch (signed=false, bundle_tor=false) is the only way to exercise it, since the job cannot run on a PR trigger. The release job is push-only and shows as skipped on a dispatch run, which is expected.

Visual changes

No UI change. Screenshots below are acceptance evidence from the packaged binary the new suite drives (see follow-up comment): the app after launch, and after the quit/relaunch with the persisted theme: light.

Security and privacy

No change. No new dependency, no new IPC channel, no runtime code touched — the app-side change is zero; everything new is test/CI/doc. The new job runs on a public repo's ubuntu-latest runner with the default read-only token, reads no secrets, and only installs the run's own artifact.

AI assistance

Written by an automated agent (Claude, via the alan PR loop). Every command quoted above was actually run in this working tree and its output read; the mutation tests exist precisely so the new assertions are not taken on trust. A human should still review the workflow job before merging.

[meinharrd]

Runner verification of the workflow change

gh workflow run release.yml --ref alan/packaged-smoke-phase1 -f signed=false -f bundle_tor=false →
run 33891141851 — success.

job	conclusion
mac-arm64	success (unsigned)
linux-arm64	success
linux-x64	success
windows-x64	success
smoke-linux-x64	success (2m47s)
release	skipped — push-only, as expected on a dispatch run

smoke-linux-x64 job — every step green:
Check out repository, Set up Node.js, Install dependencies, Install Playwright system dependencies, Download the linux x64 artifact, Resolve the expected version, Install the .deb, Smoke-test the installed .deb, Extract the AppImage, Smoke-test the AppImage, Upload the smoke-test report.

Quoted from its log:

Expecting the packaged app to report version 0.8.5-dev
Package: freedom-browser
Architecture: amd64
Version: 0.8.5~dev                  <- dpkg's mangling of 0.8.5-dev; the check is against what the app reports

[packaged] driving /opt/Freedom/freedom
  ✓  2 launch.spec.js:11 › the packaged app launches with its browser chrome (23.6s)
  ✓  3 launch.spec.js:31 › the binary under test really is a packaged Electron build (3.1s)
  ✓  4 persistence.spec.js:23 › a theme change survives a full quit and relaunch (5.6s)
  ✓  5 version.spec.js:18 › the packaged app reports the expected version (1.5s)
  5 passed (34.9s)

[packaged] driving /home/runner/work/freedom-browser/freedom-browser/squashfs-root/freedom
  ✓  2 launch.spec.js:11 › the packaged app launches with its browser chrome (3.1s)
  ✓  3 launch.spec.js:31 › the binary under test really is a packaged Electron build (3.1s)
  ✓  4 persistence.spec.js:23 › a theme change survives a full quit and relaunch (6.2s)
  ✓  5 version.spec.js:18 › the packaged app reports the expected version (1.7s)
  5 passed (15.1s)

The whole smoke job adds ~3 minutes to a release run, and no secrets are involved.

Local artifact coverage

Beyond the --dir build the task called for, I also built the real distributables locally (npm run dist -- --linux --x64) and ran the suite against the extracted AppImage, so both artifact shapes were green here before the runner saw them:

• dist/linux-unpacked/freedom → 5 passed (27.6s)
• /tmp/aitest/squashfs-root/freedom (from Freedom-0.8.5-dev.AppImage --appimage-extract) → 5 passed (24.8s)
• source, no env var: xvfb-run -a npm run test:e2e -- test-e2e/settings.spec.js → 6 passed (34.0s)
• npm test → 184 suites / 3547 tests passed, 4 suites + 15 tests skipped, 44.7s

Acceptance evidence (packaged binary, not the source build)

Freedom launched from dist/linux-unpacked/freedom by the packaged project — full chrome, tab bar, address bar, bookmarks bar, home page:

The same executable after a full quit and relaunch against the same scratch profile, still rendering the persisted theme: light (the profile was seeded with theme: dark, so this pixel state is only reachable if the change survived the restart):

Notes for review

• The persistence spec seeds theme: dark on purpose. theme: system renders as light or dark depending on the host, and explicit dark is indistinguishable from system in the DOM (applyTheme only ever sets/removes data-theme="light"), so seeding an explicit value is what makes the before/after assertion host-independent. The mutation test (relaunch against a fresh profile) shows the getSettings().theme assertion, not the DOM one, is what actually catches a persistence failure on a light-preferring host.
• smoke-linux-x64 waits on the whole linux matrix (x64 and arm64) because needs: cannot target one matrix leg. On this run that cost nothing; if arm64 ever becomes the slow leg, splitting the matrix would be the fix.
• Scope: launch, version, persistence, Linux x64 only. Bundled nodes, navigation, the headline feature, upgrade-from-previous-version and the other platforms remain manual in §6, and the playbook now says so explicitly.
• Only commit on the branch is b361ab9c, authored by this loop; there are no bot commits from anyone else on it.

[meinharrd]

[alan-review R1] — reviewed b361ab9c (the branch's only commit; no bot commits). No blocking findings. Three minors below.

What I verified

• Read the full diff (workflow, playwright config, fixtures split, 4 new spec/setup files, docs, scripts).
• Independently confirmed the dispatch run 33891141851: all jobs green, and pulled the smoke-linux-x64 job log myself — both legs (/opt/Freedom/freedom from the .deb, extracted-AppImage squashfs-root/freedom) show 5/5 passing with Expecting the packaged app to report version 0.8.5-dev. release correctly skipped (push-only).
• Reran the fixture-regression check locally on this branch: xvfb-run -a npm run test:e2e -- test-e2e/settings.spec.js → 6 passed (27.7s), so the userDataDir/electronApp fixture split is a no-op for the source-mode harness suite.
• --project=harness --list → 82 tests, zero from test-e2e/packaged/ (regex narrowing works); --project=packaged --list → preflight + 4 specs. Ran the preflight failure path with the var unset → the intended needs FREEDOM_E2E_EXECUTABLE and it is not set error, 4 did not run.
• Traced the moving parts the workflow relies on: apt-hardening.sh install passes its args straight to sudo apt-get install -y, which handles a realpath'd local .deb (and resolves its deps); freedom-linux-x64 uploads exactly one .deb + one .AppImage; the release job's pattern: freedom-* does not match smoke-linux-x64-report, so smoke traces can never leak into release assets; jest ignores test-e2e/**/*.spec.js and .setup.js matches neither harness; eslint + prettier clean on all changed files.
• src/main/index.js confirms the spec assertions: packaged Linux name is freedom, FREEDOM_TEST_USER_DATA redirects userData before anything reads it, settings live at <userData>/settings.json (what the persistence spec polls).

Minor findings (non-blocking)

M1 — launch.spec.js:44: the process.execPath === FREEDOM_E2E_EXECUTABLE assertion false-fails on symlinked or relative paths. process.execPath comes from /proc/self/exe and is fully symlink-resolved (verified locally: launching Node via a symlink reports the resolved target). The .deb this suite blesses installs /usr/bin/freedom → /etc/alternatives/freedom → /opt/Freedom/freedom (app-builder-lib's after-install.tpl update-alternatives block), so a user who follows the preflight USAGE text ("Installed packages work too") and points FREEDOM_E2E_EXECUTABLE at /usr/bin/freedom — or at a relative dist/linux-unpacked/freedom — gets a green preflight, a working launch, then expected "/usr/bin/freedom", received "/opt/Freedom/freedom" from the one spec whose comment says it trips on source checkouts. CI is unaffected (both legs use real absolute paths). Fix: compare fs.realpathSync(path.resolve(...)) of the env value, or assert the two resolve to the same file.

M2 — preflight.setup.js:50: the X_OK check is a no-op on Windows. fs.accessSync(p, fs.constants.X_OK) behaves like F_OK on Windows (no execute bit), so the "is not executable" guard only exists on Linux/macOS. Harmless today (the CI leg is Linux-only), but docs/development.md presents the suite platform-generically and later phases target other platforms — worth a one-line comment so nobody trusts that branch on win32.

M3 — a bare npx playwright test (no --project) now fails at packaged-preflight when FREEDOM_E2E_EXECUTABLE is unset. With no --project filter Playwright runs every project, so the preflight error (by design a hard fail, not a skip) turns a previously harness+live run into a failing run. Every npm script pins --project and ci.yml only uses the scripts, so nothing in-repo breaks — but a contributor running the raw command gets a confusing failure for a suite they never asked for. If that matters, test.skip() with the message when the var is unset and nothing selected the packaged project isn't cleanly expressible — a doc line in development.md ("always run e2e through the npm scripts") is the cheap fix.

For the record

b361ab9c is authored by the alan loop and has had this review; there are no other commits on the branch. The first tag-push execution of the gate (with release actually depending on smoke-linux-x64) remains unexercised until the next real release candidate, as the PR itself notes.

[meinharrd]

Landed R1's three minors in a21fc02: real-path comparison in launch.spec.js (M1), the packaged projects are registered only when FREEDOM_E2E_EXECUTABLE is set or --project packaged is passed so a bare npx playwright test stays harness+live (M3), and a comment on the Windows no-op of the X_OK check (M2).

Verified locally on a fresh npm run build -- --linux --x64: FREEDOM_E2E_EXECUTABLE=dist/linux-unpacked/freedom (relative) → 5 passed (28.5s); via a symlink /tmp/freedom-link → launch specs pass; bare npx playwright test --list lists 0 packaged tests, --project=packaged --list lists 5. Re-dispatching the workflow on this branch to re-confirm the runner leg.