QA that cannot cheat.
Traditional test suites lie politely. A CSS selector can “pass” while the button is invisible. Mocks skip the network you actually ship. Assertions read the DOM, not what a human saw. Headless bots often get a different site than your users.
Nightshift is an AI QA intern for real apps. It opens a Solari stealth browser, drives a computer-use loop toward your goal, and only marks PASS when a vision step sees the success state on a real screenshot — and a session recording exists to prove it. No recording → cheated-blocked. Selectors are never the pass condition.
Built by Yahya Qureshi (yayaq1) for Pinetree Research / Solari’s SWE intern challenge.
- Founders who ship weekly and need proof staging still works
- Yahya, running Cursor community apps and his own products
- Harry — agents that finish what they start, reliability over intelligence, recordings, persisted logins, work at 2am
This is not a GitHub-repo reviewer, not a scraper, and not Playwright-with-a-wrapper. Solari’s marketing site already lists a generic “QA Test Runner.” Nightshift is different on purpose: computer-use + vision judging + anti-cheat rules + replay as the artifact.
- Pass = visible success. Screenshots + vision (or a clearly labeled heuristic fallback). Never selectors.
- Real URL. No network mocks.
- Stealth browser. Solari stealth + proxy + captcha so you test the bot-protected site users hit.
- Recording mandatory.
recording: trueon launch. A pass without a replay is invalid. - Persistent profiles. Login once, save
storageState, reuse — no re-typing 2FA every run. - 2am re-run. Same flow, same profile, compare finished vs not.
| Primitive | Package | Nightshift use |
|---|---|---|
| Browser | @solarisdk/browser |
Stealth + recording computer-use loop |
| Sandbox | @solarisdk/sandbox |
Score the run, diff steps, write the report |
| Desktop | @solarisdk/desktop |
Native/GUI path or browser fallback |
Docs: docs.getsolari.com · API: api.getsolari.com
Redeem STARTER1MO-MKY4BNDK at console.getsolari.com.
npm install
cp .env.example .env.local # optional: SOLARI_API_KEY, OPENAI_API_KEY
npm run devOpen http://localhost:3000.
- Paste a URL (default: the built-in
/fixturesignup → empty state) - Optional: “what should work”, test-account credentials, desktop-app toggle, 2am re-run
- Run nightshift → shareable
/run/[id]with steps, screenshots, anti-cheat verdict, environments
If SOLARI_API_KEY is missing, the UI still runs a clearly labeled simulated vision run. It never pretends Solari session metrics are live.
export SOLARI_API_KEY=slr_live_...
# optional vision judge:
export OPENAI_API_KEY=...
npm run devLive runs: stealth+recording browser → per-step screenshot judge → sandbox report → desktop fallback if the browser cannot finish.
export SOLARI_API_KEY=slr_live_...
npm start -w nightshift-qa-tsapps/web Next.js App Router product
packages/runner Solari orchestration + anti-cheat judge
examples/nightshift-qa-ts Minimal TS script (browser + sandbox)
Runs are stored as JSON under .data/runs/. Screenshots land in apps/web/public/runs/.
- Launch stealth+recording browser; attach/create a profile when credentials exist
- Computer-use loop: screenshot → decide click/type/scroll → act → hard step cap
- After each step, judge visible UI against the goal (LLM if keyed, else heuristic fallback)
- Final PASS only if success is visible and a recording session id exists
- Desktop path when “desktop app” is checked or the browser cannot proceed
- Sandbox writes the markdown report and score
Always await solari.close() in TS or the process hangs. Poll ~30s for replay after release.
GitHub write APIs (fork / rename wifi-tester / create nightshift) were 403 for the cloud agent token, so Nightshift ships in this repo (yayaq1/TSS_playground) on branch cursor/nightshift-26d5 as a standalone app replacing the disposable playground.
MIT · Built on Solari · Finish what you start.