test: centralized/DI test suite (unit + functional + E2E) + fast-tier CI - #1
Merged
Merged
Conversation
Move the pure logic inlined in setup.mjs — CDN listing parse + numeric version sort, SV3 zip selection, node-version check, file-age formatting, hashes-filename derivation, and sqlite3 target paths — into an exported, unit-testable scripts/lib/sv3.mjs. Rewire setup.mjs to import them and remove every inline copy, including a duplicated node-version check in the --status block. Add 7 node:test parity tests. Behavior-preserving: setup.mjs --help/--status output is byte-for-byte identical to pre-refactor. The hashes-filename derivation bug is preserved verbatim here and fixed separately in card sv3-runner-xxl.3. Card: sv3-runner-xxl.1 Authored by: Aaron Lippold<lippold@gmail.com>
Rename the package to @mitre/sv3-runner (kept private) and replace the broken `npx playwright test` default with scoped node:test scripts. `test`, `test:unit`, and `test:functional` use `node --test 'test/**/*.test.mjs'` globs so the suite covers only test/ and never the extracted sv3-app/ — bare `node --test` otherwise runs 27 of DISA's bundled app tests. test:e2e (Playwright) is wired in card .6. Card: sv3-runner-xxl.12 Authored by: Aaron Lippold<lippold@gmail.com>
Establish code standards for the v2 test-suite work (sv3-runner-tu5.1). No linter/formatter existed before this. - eslint.config.mjs: flat/ESM config — @eslint/js recommended + Node globals; eslint-config-prettier/flat last so ESLint owns code quality and Prettier owns formatting (kept as separate tools). - .prettierrc.json: singleQuote + printWidth 100. - .prettierignore: excludes generated/vendored dirs (node_modules/, sv3-app/, downloads/). - package.json: `lint` (eslint + prettier --check) and `format` scripts; eslint/@eslint/js/globals/eslint-config-prettier/prettier devDeps. Authored by: Aaron Lippold<lippold@gmail.com>
Bring existing sources into compliance with the new Prettier gate (sv3-runner-tu5.1). Formatting only — no behavior change: arrow-param parens, long-line wrapping, trailing commas. Verified: setup.mjs --help and --status output byte-identical to pre-change; parity tests 7/7. Authored by: Aaron Lippold<lippold@gmail.com>
One source of truth for shelling out (sv3-runner-tu5.2). setup.mjs defined run()/runCapture() inline and run.mjs had its own execSync — a DRY violation and the reason a third copy nearly appeared. This is the injectable dependency the pipeline (tu5.3) will consume. - scripts/lib/exec.mjs: exported run (echoes cmd, stdio inherit, throws on failure) + runCapture (trimmed stdout, null on non-zero — the soft probe). Project root derived from the module's own location, so the default cwd is unchanged. - setup.mjs/run.mjs import the shared runner; inline copies deleted (execSync now appears only in exec.mjs). - test/unit/exec.test.mjs: real-command unit tests (no mocks) pinning success/trim, null-on-failure, echo, and throw-on-failure. Behavior-preserving: setup.mjs --help/--status byte-identical to before. run.mjs launch now prints a ` $ npx electron ...` trace line — run()'s defined "logs the command" contract, matching setup.mjs's convention. Authored by: Aaron Lippold<lippold@gmail.com>
- .gitignore: exclude per-machine session-recovery files (.beads/recovery-context.md, recovery-prompt.md, archive/) so they stop showing as untracked. They are local convenience, not shared — the shared card history lives in the Dolt DB and interaction log. - .beads/interactions.jsonl: bd's tracked interaction log — accumulated status-change events (tu5.1/tu5.2 closes and the superseded xxl epic). Card creations are not logged here, so the new npm-audit card (sv3-runner-3es) lives only in the Dolt DB, not this file. Authored by: Aaron Lippold<lippold@gmail.com>
…mjs) Turn the inline extract→install→build→patch flow into dependency-injected functions so every shell-out is unit-testable by asserting the exact command with a spy — no mocks, no real rebuild (sv3-runner-tu5.3). - scripts/lib/pipeline.mjs: extractApp/installDeps/buildSqlite each take an injected `run` (default = exec.run); patchSqlite is pure-fs and reuses lib/sv3.mjs's sqlite3TargetDir/sqlite3BinaryPath (no duplicated path logic). ExtractError classifies decompress failures so the caller prints unzip-specific guidance while other failures surface unchanged. - scripts/setup.mjs: main() now composes the steps; idempotency skip-checks and --force gating unchanged. - scripts/run.mjs: launch extracted to launchSv3(). - test/unit/pipeline.test.mjs: asserts the exact command each shell-out step issues (spy), a real-fs patchSqlite test, and the ExtractError path. Behavior-preserving: setup.mjs --help/--status byte-identical to prior HEAD; a real `npm run setup --force` ran all four steps to completion and `npm start` launched SV3 to its main window (sqlite3 loaded). Authored by: Aaron Lippold<lippold@gmail.com>
Authored by: Aaron Lippold<lippold@gmail.com>
Drive the real scripts as child processes against isolated temp sandboxes and assert their observable CLI contract (sv3-runner-tu5.5). Since setup.mjs/run.mjs derive PROJECT_DIR from their own location, each test copies scripts/ into a fresh mkdtemp sandbox and spawns from there — so PROJECT_DIR is the sandbox, never the real repo. No network, no download, no native rebuild, no mutation of the working tree. Covers: --help usage/exit, --status across no-zip/zip/extracted/binary states, --clean removal (temp fixture, scripts untouched), the idempotency skip branches, and run.mjs's non-zero exit + guidance when sv3-app/ is missing. Every assertion pins a specific exit code and exact message. Authored by: Aaron Lippold<lippold@gmail.com>
Authored by: Aaron Lippold<lippold@gmail.com>
The original inline `.replace(/-/g, '-')` was a no-op, producing `U_STIGViewer_-3-7-0_Hashes.txt` — which 404s on the DISA CDN, so the hashes download silently failed (sv3-runner-tu5.4). Verified against the live CDN: the real manifest is version-keyed and platform-independent — `U_STIGViewer_3-7-0_Hashes.txt` (HTTP 200), shared by the linux_x64 and win32_x64 zips of a version. The fix extracts the trailing version with an ANCHORED `/(\d+-\d+-\d+)\.zip$/` (an unanchored match would grab the "64" from "x64") and builds `U_STIGViewer_<v>_Hashes.txt`. parity.test.mjs's hashesFilenameFor assertion, which had pinned the old buggy value as a characterization, is updated to the correct name. Authored by: Aaron Lippold<lippold@gmail.com>
Fuller boundary coverage for the kept lib/sv3.mjs helpers, beyond the parity characterization (sv3-runner-tu5.4): - parseSv3Listing against a realistic CDN listing fixture (excludes win32/msi/hashes, keeps only linux_x64 zips) - selectLatestZip numeric ordering incl. 3-10-0 > 3-7-0 - selectSv3Zip: CLI arg > newest in downloads/ > default - checkNodeVersionOk inclusive [min,max] boundaries - fileAgeLabel minute/hour/day bands + boundaries - sqlite3TargetDir/BinaryPath across darwin-arm64/x64, linux-x64, win32-x64 - hashesFilenameFor pinned to the researched correct DISA name (this test, written first, is what drove the fix and caught a wrong intermediate attempt) test/fixtures/cdn-listing.html mirrors the real DISA directory listing. Authored by: Aaron Lippold<lippold@gmail.com>
`npm run download` writes the DISA hash manifest next to the zip in downloads/, but only *.zip was ignored — the manifest showed as untracked. Matches the documented intent (zip + hashes are generated artifacts). Authored by: Aaron Lippold<lippold@gmail.com>
Authored by: Aaron Lippold<lippold@gmail.com>
…ight _electron) The acceptance test that justifies the project (sv3-runner-tu5.6): launch the extracted STIG Viewer 3 via Playwright's _electron with a .cklb path in argv (the mechanism run.mjs/main.js:166 use), then assert real SV3 renders the checklist's rules with their exact status and comment — proving a CKLB is accepted by the real STIG Viewer (heimdall2 ADR-003). - @playwright/test devDependency; playwright.config.mjs (testDir test/e2e); `test:e2e` script; playwright.config.mjs added to the lint/format gate. - test/e2e/cklb.spec.mjs: launches sv3-app (--no-sandbox, fixture path), selects the main window (dist/index.html, not the splash), asserts a rule renders not_a_finding, another open, a third not_applicable (differing statuses prove SV3 reads the field), then selects the first and asserts its comment textarea shows "hi". Screenshots, closes cleanly, skips if sv3-app/ absent. - test/fixtures/sample.cklb: minimal fixture derived from a real SV3-produced RHEL 10 checklist (3 real rules). Verified: `npx playwright test` green; screenshot shows the rendered rule with its status + comment; no leaked Electron process; node:test suite unaffected. Authored by: Aaron Lippold<lippold@gmail.com>
Authored by: Aaron Lippold<lippold@gmail.com>
The always-on gate that runs on every push and pull_request (sv3-runner-tu5.7): Node 22, `npm ci --ignore-scripts`, `npm run lint`, and the scoped `node --test 'test/**/*.test.mjs'` (unit + functional). --ignore-scripts skips the Electron binary download, the sqlite3 native build, and the Playwright browser download — none are needed by lint or the node:test suite — keeping this tier fast and network-light. The real setup/rebuild/Electron path is the cross-platform matrix (tu5.8). Uploads lint/test logs as an artifact on failure. Current action majors (checkout@v7, setup-node@v6, upload-artifact@v7). Authored by: Aaron Lippold<lippold@gmail.com>
There was a problem hiding this comment.
Pull request overview
Establishes a centralized, dependency-injected test suite (unit + functional + E2E) for sv3-runner, while refactoring the CLI scripts to be testable-by-conposition (shared exec + DI pipeline) and wiring a fast-tier CI job to run lint + node:test.
Changes:
- Extracted reusable, unit-testable helpers and pipeline steps into
scripts/lib/*(centralexec, DIpipeline, puresv3helpers) and rewiredsetup.mjs/run.mjsto use them. - Added a full test pyramid: characterization + edge-case unit tests, sandboxed functional CLI tests, and a Playwright Electron E2E acceptance test using a CKLB fixture.
- Added ESLint (flat config) + Prettier gates, updated npm scripts, and introduced a “fast tier” GitHub Actions workflow running on Node 22 with
--ignore-scripts.
Reviewed changes
Copilot reviewed 20 out of 22 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| test/unit/sv3.test.mjs | Adds edge-case unit coverage for pure SV3 helper functions (listing parsing, version selection, hashes filename, sqlite paths). |
| test/unit/pipeline.test.mjs | Unit tests DI pipeline steps by asserting exact shell commands and verifying filesystem patching behavior. |
| test/unit/parity.test.mjs | Characterization/parity tests pin helper behavior to pre-extraction outputs to ensure refactor preserves semantics. |
| test/unit/exec.test.mjs | Pins the behavior contract for the centralized run/runCapture shell runner. |
| test/functional/cli.test.mjs | Functional CLI contract tests using an isolated temp sandbox to avoid mutating the repo or hitting the network. |
| test/fixtures/sample.cklb | CKLB fixture for the E2E acceptance test (used to validate SV3 renders rule statuses/comments). |
| test/fixtures/cdn-listing.html | Realistic CDN directory listing fixture for unit tests. |
| test/e2e/cklb.spec.mjs | Playwright Electron E2E acceptance test launching the real SV3 app and asserting CKLB rendering. |
| scripts/setup.mjs | Refactors setup flow to thin composition over shared exec + DI pipeline + pure helpers; fixes hashes filename derivation via helper. |
| scripts/run.mjs | Refactors launch path to use shared exec runner and encapsulates launch in a helper function. |
| scripts/lib/sv3.mjs | Introduces pure helper module (listing parsing, zip selection, version checks, hashes filename derivation, sqlite path helpers). |
| scripts/lib/pipeline.mjs | Introduces DI pipeline module for extract/install/build/patch steps with explicit ExtractError classification. |
| scripts/lib/exec.mjs | Introduces centralized command runner (run / runCapture) as the single shell-out implementation. |
| playwright.config.mjs | Adds Playwright configuration for Electron E2E (single worker, timeout, output dir). |
| package.json | Updates package name, defines test pyramid scripts, adds lint/format scripts, and adds devDependencies for linting and Playwright. |
| package-lock.json | Locks new devDependencies and engine constraints consistent with the Node 22 fast-tier CI. |
| eslint.config.mjs | Adds ESLint flat config for Node ESM + integrates eslint-config-prettier. |
| .prettierrc.json | Adds Prettier formatting configuration. |
| .prettierignore | Ignores generated/vendored directories from formatting. |
| .gitignore | Ignores downloaded hash manifests and local .beads/ recovery artifacts. |
| .github/workflows/ci.yml | Adds fast-tier workflow (Node 22, npm ci --ignore-scripts, lint, node --test, logs on failure). |
| .beads/interactions.jsonl | Adds Beads interaction log entries documenting issue status changes related to this work. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Authored by: Aaron Lippold<lippold@gmail.com>
Add the legal and community files so the public MITRE repo is properly licensed and contributor-ready: - LICENSE: verbatim Apache-2.0 (resolves GitHub license detection) - NOTICE: MITRE copyright (Case 18-3678) + note that SV3 is a USG/DISA product this project does not vendor or redistribute - CONTRIBUTING, SECURITY, CODE_OF_CONDUCT: adapted from MITRE SAF standard templates (saf-security@mitre.org channel, Contributor Covenant 2.1) - .github issue/PR templates + @mitre/saf CODEOWNERS - package.json: license/author/repository/bugs/homepage (private:true kept, name unchanged) - README: License section references LICENSE and NOTICE Authored by: Aaron Lippold<lippold@gmail.com>
The bd interaction log is chatty per-session work-tracking metadata that does not belong in a public repo (issues sync via Dolt refs, not this file). Untrack it and add it to .gitignore; the working copy is preserved locally for bd. Authored by: Aaron Lippold<lippold@gmail.com>
Node 24 is the project's supported toolchain, matching Electron 40's bundled Node (24.x). The prior 'Electron 40.1.0 native-module ABI requires Node 22' rationale was empirically false: under Node 24, npm run setup (extract + rebuild sqlite3 + patch), the full node:test suite (34/34), and the Playwright _electron CKLB launch all pass. - package.json engines >=24.0.0 <25.0.0; .mise.toml node=24 - scripts/setup.mjs MIN/MAX_NODE_MAJOR=24; version-mismatch message no longer claims an ABI requirement - .github/workflows/ci.yml node-version 24 - README / SECURITY / CONTRIBUTING Node refs + false-ABI wording corrected - checkNodeVersionOk boundary tests updated to the 24 policy Authored by: Aaron Lippold<lippold@gmail.com>
Audit fixes after several statements were found wrong or stale against the actual code/CDN: - DISA ships linux_x64 AND win32_x64 builds (not linux-only); no macOS build - project is the two scripts + scripts/lib/ + a test/ suite (not 'two scripts') - npm test is node:test, not playwright; the test suite exists (was documented as not-yet-implemented) - sqlite3-offline-next ships linux-x64/darwin-x64/win32-x64/win32-ia32 prebuilts, never darwin-arm64 (the reason we rebuild on Apple Silicon) - tracked-files list and the Node version section brought current - removed the false 'CDN may require VPN' note (CDN is public, verified 200) Authored by: Aaron Lippold<lippold@gmail.com>
…s/windows Heavy CI tier (e2e.yml): on ubuntu/macos/windows, run the real setup (download SV3 from the DISA CDN -> extract -> @electron/rebuild sqlite3 -> patch), assert the native binary landed (verify:setup), then launch SV3 and open a fixture .cklb via Playwright _electron. Linux runs under xvfb. - scripts/verify-setup.mjs: integration-smoke that derives the binary path from lib/sv3.mjs (sqlite3BinaryPath) so it can't drift from where setup writes it; exits non-zero with setup guidance if the binary is absent - test/helpers/sandbox.mjs: shared sandbox harness extracted from cli.test so it and verify-setup.test share one source (DRY) - package.json: verify:setup script - Action majors verified live: checkout@v7, setup-node@v6, cache@v6, upload-artifact@v7 Authored by: Aaron Lippold<lippold@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test suite + publish-readiness (v2)
Establishes a centralized, testable-by-construction test suite for sv3-runner and wires up CI.
Architecture
scripts/lib/exec.mjs— one centralized command runner (run/runCapture); no duplicated shell-out logic.scripts/lib/pipeline.mjs— the setup flow (extract → install → build → patch) as dependency-injected steps, so every shell-out is unit-tested by asserting the exact command with a spy (no mocks).setup.mjsis now thin composition;run.mjslaunch islaunchSv3().scripts/lib/sv3.mjs— pure helpers (kept from the extraction), now fully edge-case covered; fixed thehashesFilenameForbug (was a no-op.replacethat 404'd the DISA hash manifest).Test pyramid
test/unit/) — pure helpers + DI command assertions.test/functional/) — drivesetup.mjs/run.mjsas child processes in isolated temp sandboxes (--help/--status/--clean/idempotency/run-missing), no network, no mutation of the tree.test/e2e/) — Playwright_electronlaunches the real STIG Viewer 3 and opens a CKLB fixture, asserting rules render with their exact status + comment (the Heimdall2 CKLB-acceptance test).Tooling & CI
node --test 'test/**/*.test.mjs'..github/workflows/ci.yml— fast tier on every push/PR: Node 22,npm ci --ignore-scripts, lint, node:test. The cross-platform matrix + real setup/Electron E2E follow in a later card.This PR run exercises the fast-tier workflow.