Proposal: Reproducible stealth benchmark harness in CI

[JPeetz]

Proposal: Reproducible stealth benchmark harness in CI

Summary

STEALTH_TESTS.md is a valuable artifact — it's the single clearest piece of evidence in the repo for the project's core claim. Right now it's a hand-authored snapshot: one date (2026-02-10), one machine, one Chrome version, one platform. That format has a natural half-life. Chrome ships every ~4 weeks, nodriver reacts, detection vendors update rules, and there's currently no way to notice a regression until a user reports it (see e.g. #22, which is exactly this class of issue: a silent stealth capability broke and only surfaced through a user hitting it).

This proposal is to turn STEALTH_TESTS.md from a snapshot into a CI-run harness that produces machine-readable results on a schedule and generates the markdown table from the latest runs.

I'm happy to draft the harness as a PR if this direction is welcome — filing as an issue first to check alignment before writing code.

Motivation

• Regression detection. WebGL Renderer spoofing impossible because Page.addScriptToEvaluateOnNewDocument silently no-ops without Page.enable #22 (WebGL renderer spoofing broken by Page.addScriptToEvaluateOnNewDocument no-op without Page.enable) is the pattern this would catch. A daily/weekly canary catches these before they reach users.
• Credibility. The current COMPARISON.md claims are hard to defend without reproducible numbers. A public CI dashboard is a much stronger position than "trust the maintainer's laptop."
• Multi-platform coverage. Windows-only, headed-only, single-Chrome-version testing misses failure modes that only appear on Linux headless (common deployment target for MCP servers in containers), on arm64, or on newer Chrome builds.
• TLS/JA3 fingerprinting is the biggest unaddressed layer. nodriver handles the JS layer beautifully. TLS fingerprint is a separate detection surface that isn't tested at all today.

Proposal

Phase 1 — structured results + CI

• tests/stealth/ suite using pytest + nodriver.
• Each detection target is a test module returning a structured result: PASS / FAIL / DEGRADED / UNREACHABLE, plus the raw fingerprint payload as a pytest artifact.
• Results written to docs/stealth-results/YYYY-MM-DD-<runner>.json.
• Existing STEALTH_TESTS.md regenerated from the latest N JSON files (script in scripts/render_stealth_report.py).
• GitHub Actions workflow: scheduled weekly + on-demand (workflow_dispatch).

Phase 2 — expanded target matrix

Keep everything currently tested (nowsecure.nl, CreepJS, Sannysoft, Intoli), and add:

• Fingerprint surface: browserleaks.com (WebGL, canvas, audio, fonts, TLS), amiunique.org, pixelscan.net.
• TLS/JA3/JA4: tls.peet.ws/api/all, check.ja3.zone — records the TLS fingerprint per Chrome version. This is the layer most stealth libraries silently fail on.
• Public vendor demo pages (where they exist and are explicitly public): DataDome demo, Akamai bot-detection demo, PerimeterX/HUMAN demo. Testing against public vendor demos is standard practice and stays inside their intended use — we would not test against live customers of these vendors.

Phase 3 — matrix build

• Runners: ubuntu-latest (x64), macos-latest (arm64), windows-latest.
• Modes: headed + headless=new.
• Chrome versions: latest stable + one previous major (via puppeteer's installer or Playwright's playwright install chromium@…).

Phase 4 — canary + auto-issue

Separate scheduled workflow running against latest Chrome + latest nodriver. When the harness regresses (a target that was PASS drops to FAIL), the workflow opens an issue automatically with the failing target, the diff in fingerprint payload, and links to the run logs.

What this doesn't propose

• Changing the framing of COMPARISON.md (that's a separate conversation for a separate issue, if at all).
• Testing against live customers of detection vendors — only public demo endpoints, only where explicitly public.
• Reducing capability to pass tests — the harness measures, it doesn't gate.

Language / accuracy note (small)

While drafting this I noticed one specific line in the current STEALTH_TESTS.md that could be tightened: the X.com section describes "logged-out profile access past a login modal," which is real and useful, but the surrounding text implies Arkose/FunCaptcha was defeated. FunCaptcha wasn't in the flow tested. I'd suggest rewording that section to describe exactly what was demonstrated (login-wall scraping) without the Arkose framing. Happy to include this in the same PR.

Willing to do the work

If this direction is welcome, I'll open a PR with Phase 1 (structured results + minimal GitHub Actions workflow reproducing the existing test targets) as a starting point, and iterate on Phase 2+ in follow-ups. Wanted to sanity-check the approach with you before writing code.

Thanks for maintaining this — the network hook system and the CDP-level cloning are genuinely differentiated work, and the operational hygiene in recent commits (idle reaping, orphan profile cleanup, stdout pollution fix, bearer auth) is well above average for a project this age.

[vibheksoni]

Thanks for taking the time to put together such a detailed proposal I really appreciate the thought that went into it.

I’ve assigned this issue to myself. I’m currently away from home, but I’ll read through everything in detail once I’m back and give it the attention it deserves. At first glance it looks like a solid direction, and I’ll follow up after I’ve had a chance to review it properly.

[JPeetz]

Thanks, appreciate you flagging it and no rush at all — enjoy the time away. I'll start prototyping Phase 1 (structured results + minimal GH Actions workflow) in a scratch repo so I can share working code alongside the discussion when you're back.