Modern browser automation as an engineered product: a repo-owned controlled target, strict TypeScript, typed API clients, Zod contracts, reusable fixtures, visual baselines, Axe accessibility checks, sharded CI, observability, and report-grade failure diagnostics.
| Evidence | Link |
|---|---|
| Live report | Allure history |
| Release | v1.0.0 |
| CI | |
| Repository activity | Default-branch commits · Pull requests |
| Docs and assets | Documentation · Screenshots |
| Best failure evidence | Failure example |
| Engineering question | Implemented answer |
|---|---|
| How can broad browser coverage stay deterministic? | A repo-owned Conduit target, stable seed data, explicit project boundaries, and pinned browser images. |
| How are API and UI layers kept coherent? | Typed API clients, Zod validation, API-driven setup, and page objects share controlled contracts. |
| How are failures made actionable? | Screenshots, traces, HTML and Allure reports, redacted logs, runtime metrics, and documented triage. |
| How is flakiness governed? | Project-specific retry rules, a retry budget, expiring quarantine metadata, and scheduled analytics. |
| How does CI scale? | Linux and Windows jobs, UI and visual shards, browser smoke projects, and machine-readable metrics. |
Release v1.0.0 includes cross-platform line-ending enforcement, project-specific
retry policy, machine-readable runtime and retry metrics, expiring quarantine
governance, OpenTelemetry support, and explicit controlled-target limitations.
See CHANGELOG.md.
Run the full local verification gate:
npm run verify:targetThe gate runs:
npm run format:checknpm run check:runtimenpm run lintnpm run type-checknpm run check:secretsnpm run check:anti-patternsnpm run check:visual-snapshotsnpm run check:openapi-contractnpm run with:target -- npm run check:envnpm run test:apinpm run test:e2enpm run test:visualnpm run test:accessibilitynpm run test:contractsnpm run test:cross-browser- scheduled
npm run test:cross-browser:full
CI enforces formatting, linting, type checking, secret scanning, anti-pattern scanning, visual snapshot hygiene, OpenAPI/Zod contract alignment, environment readiness, API tests, sharded authenticated and anonymous UI tests, sharded visual tests, Axe accessibility tests, selector-contract tests, and cross-browser smoke tests against the local controlled target. Linux jobs run inside the pinned mcr.microsoft.com/playwright:v1.60.0-noble image. Report artifacts include Allure results, JUnit XML, JSON results, and Playwright HTML output. The visual job runs on Windows to match the committed win32 Chromium baselines.
The local target lives in test-target/ and implements the Conduit API/UI contract used by the tests. It starts with deterministic seed credentials:
npm run target:start
npm run with:target -- npm run check:env
npm run with:target -- npm run test:api
npm run verify:targetDefault target values are injected only by the target wrapper and are fixture-only values:
- base URL:
http://127.0.0.1:4300 - API URL:
http://127.0.0.1:4300/api - seed user email:
seed.user@example.test - seed user password: fixture value generated by
scripts/target-runner.js - seed username:
seeduser
Use npm run target:seed while the target is running to reset data to the known baseline.
- API tests validate authentication, article, comment, profile, and feed contracts through typed API clients.
- E2E tests cover focused user workflows through page objects and explicit navigation.
- Visual tests compare stable Chromium screenshots against committed baselines under
tests/visual. - Accessibility tests scan critical authenticated and unauthenticated pages with Axe.
- Authenticated and anonymous browser projects are separated in Playwright config so login tests do not inherit saved user state.
flowchart TD
Config["Zod config"] --> BaseApi["Base API client"]
Logger["Redacted logger"] --> BaseApi
Builders["Test data builders"] --> ApiClients["API clients"]
BaseApi --> ApiClients
ApiClients --> Fixtures["Playwright fixtures"]
Pages["Page objects"] --> Fixtures
Fixtures --> Specs["API, E2E, visual specs"]
Specs --> Reports["HTML and Allure reports"]
src/api/ Typed API clients and Zod models
src/builders/ Deterministic test data builders
src/fixtures/ Playwright fixtures for auth, API clients, scenarios, observability, and page objects
src/observability/ Optional OpenTelemetry helpers and Playwright step wrappers
src/pages/ Page objects and reusable components
src/utils/ Config, logging, and custom matchers
tests/api/ API contract checks
tests/accessibility/ Axe accessibility checks
tests/contracts/ Selector contract checks for controlled UI test IDs
tests/e2e/ Browser workflows
tests/setup/ Report-visible setup and teardown projects
tests/visual/ Visual regression checks and baselines
test-target/ Repo-owned Conduit-compatible API and UI target
docs/adr/ Architecture decision records
docs/openapi/ Controlled target OpenAPI contract source
Use Node 20 or newer, with Node 20 listed in .nvmrc and .node-version as the baseline runtime.
npm ci
npx playwright install chromium
cp .env.example .envFor normal local verification, prefer npm run verify:target instead of hand-maintaining .env. For exploratory external runs, update .env with credentials for a test account and explicit target URLs. Do not commit .env, .auth, reports, traces, screenshots, or generated browser artifacts.
The controlled UI exposes stable data-testid values for repeated or test-critical surfaces such as article cards, article body, article author, article title, article description, article date, sidebar tag lists, comments, validation errors, profile links, and profile fields. Page objects centralize these IDs in src/pages/test-ids.ts and otherwise prefer roles, labels, placeholders, and scoped locators.
Fixture boundaries (anonymous vs. authenticated API clients) are documented in Writing tests.
OpenTelemetry is opt-in for local runs:
npm run observability:up
npm run test:otel
npm run observability:downOTEL_ENABLED=true starts per-test root spans, observed Playwright step spans, API request spans, and cleanup spans. Each test result includes an otel-trace-context.txt attachment with the trace and span identifiers. Jaeger is available at http://127.0.0.1:16686 when the observability profile is running.
Run npm run flake:report after a Playwright JSON result is available to write
test-results/flake-report.md and test-results/portfolio-metrics-v1.json.
CI enforces the retry budget with npm run flake:check on pull requests and scheduled runs. See
Flakiness policy.
- Playwright HTML:
playwright-report/ - Allure results:
allure-results/ - JUnit XML:
test-results/junit.xml - JSON results:
test-results/results.json - Portfolio metrics:
test-results/portfolio-metrics-v1.json
| Surface | Matrix or shards | Retry policy |
|---|---|---|
| API | 1 Linux job | 0 |
| Authenticated and anonymous Chromium | 2 Linux shards | 1 UI retry |
| Visual Chromium | 2 Windows shards | 0 |
| Accessibility and selector contracts | 1 Linux job each | 0 |
| Browser smoke | Firefox, WebKit, mobile Chromium | 1 UI retry |
Current workflow duration and sample size are published on the portfolio dashboard.
The CI workflow publishes Allure history from green default-branch controlled-target results to the gh-pages branch. The manual Allure Report Deploy workflow can republish the same report path on demand.
This project is licensed under the MIT License. When cloning, forking, redistributing, or deriving work from this repository, preserve the license notice and attribute the original source to Prayag Vyas (prayagv) as described in NOTICE.
- Latest CI workflow: GitHub Actions CI
- Published report: Allure history
- High-signal tests:
tests/api,tests/e2e/article-lifecycle.spec.ts,tests/contracts/selectors/controlled-ui.selectors.spec.ts,tests/visual, andtests/accessibility - Local proof command:
npm run verify:target - Verification evidence: Verification evidence
- No untracked publishable files; every accepted
tests/visual/**/*-snapshots/*.pngbaseline is listed intests/visual/visual-snapshots.manifest.json. - No ignored reports, traces, screenshots, videos,
.auth, or local environment files committed. npm run verify:targetis green on Node 20.- GitHub Actions is green against the controlled target on the public default branch.
- CI and Allure badges point to the repository workflow and GitHub Pages report URLs.
- Confirm at least one meaningful initial commit exists before publishing.
For deterministic CI, use the repo-owned controlled target or point BASE_URL and API_URL at a controlled RealWorld-compatible deployment, not a public demo service. See Controlled Test Environment. The checked-in Docker Compose harness runs the repo-owned Conduit-compatible target on port 4300; set TEST_USER_PASSWORD explicitly before starting it.
See Commands for the full npm script reference and Writing tests for fixture and test-authoring design notes.
- User accounts cannot be deleted through the public API, so tests use unique generated users and clean up deletable resources such as articles/comments.
- Generated reports are intentionally ignored by Git; publish only clean CI artifacts from green runs.
- The Docker Compose harness runs the repo-owned Conduit-compatible target for deterministic local checks. External RealWorld-compatible deployments remain supported through explicit
BASE_URL,API_URL, and seeded-user environment variables. - User retention and cleanup rules are documented in Data isolation and retention.
- Architecture
- Commands
- Configuration guide
- Debugging test failures
- Dos and don'ts
- Execution guide
- Observability
- Writing tests
- Portfolio review guide
- Engineering history
- Flakiness policy
- Flake report example
- Failure example and triage
- Offline failure evidence bundle
- Seeded defect examples
- Contributing guide
- Controlled test environment
- Data isolation and retention
- Test strategy matrix
- Verification evidence
- Controlled target OpenAPI contract
- ADR-001: Playwright over Cypress
- ADR-002: Conduit as Target
- ADR-003: Hybrid API + UI Strategy
- ADR index