Skip to content

ci: the BrowserAdapter integration suite is skipped on every CI run, so its regressions only surface locally #78

Open
Open
ci: the BrowserAdapter integration suite is skipped on every CI run, so its regressions only surface locally#78
Labels
area/cienhancementNew feature or requestready-for-agentsize/2h

Description

@ivndev001
ivndev001
opened on Sep 11, 2026

What is wrong

CI never runs the BrowserAdapter integration suite. .github/workflows/ci.yml's test step sets:

- name: test
  env:
    SKIP_BROWSER_INTEGRATION: "1"

with the stated reason:

bun test runs test files in parallel by default. The low-level BrowserAdapter integration suite
flakes under the runner's parallel 2-vCPU load (its own Bun.serve fixture refuses connections) —
skip it here; the e2e suite covers the real browser→findings path in CI. It runs in full locally.

The diagnosis is right: the flake is parallel load, not the suite. But the remedy throws the suite
away on every CI run rather than removing the contention, so a regression in the adapter's own
seam only ever surfaces on someone's laptop.

Why now, specifically

The reason to skip it was partly budget: the job could not afford a slow serial suite. That
constraint is gone. As of #77 the job runs in roughly 35 seconds end to end (it previously
timed out at 20 minutes on the Chromium dependency install), against a timeout-minutes: 20
ceiling. There is now ample room to run this suite properly.

What to do

Run the browser integration suite in CI without the parallelism that makes it flake, rather
than skipping it — for example by giving that suite its own serial test invocation while the rest
of the suite keeps running in parallel. Do not simply drop SKIP_BROWSER_INTEGRATION and hope: the
contention it works around is real, and re-introducing a flaky required check is worse than the
current honest gap.

Whatever shape you choose, the fixture's port binding should not depend on tests not racing each
other — a fixture that only works when nothing else is running is the underlying fragility.

Acceptance criteria

  • the BrowserAdapter integration suite executes in CI — it is no longer skipped wholesale
  • it passes on three consecutive runs on the 2-vCPU runner, evidenced by run links in the PR.
    One green run does not distinguish a fix from a lucky scheduling order
  • the rest of the test suite still runs in parallel — the fix must not serialise everything
  • the job stays comfortably inside timeout-minutes: 20; state the new wall-clock in the PR
  • no test is weakened, deleted or marked skipped to achieve the above

Notes

Related but distinct: #58 is about a unit test launching a real browser to assert on file
contents. This issue is about the integration suite being excluded from CI entirely.

Found while repairing this repo's CI in #77 (developerz.ai#3400).

Activity

  1. developerz-ai
    added
    enhancementNew feature or request
    on Sep 11, 2026
  2. sebyx07
    added
    area/ci
    size/2h
    on Sep 11, 2026

  3. developerz-ai commented on Sep 11, 2026

    @developerz-ai
    developerz-aiBot
    on Sep 11, 2026 – with Developerz AI
    Contributor

    Scoped: Run BrowserAdapter Integration Suite in CI

    Problem: The BrowserAdapter integration suite is skipped in CI via SKIP_BROWSER_INTEGRATION=1 due to parallel execution causing flaky failures (Bun.serve fixture refuses connections under 2-vCPU load). This leaves the suite untested in the pipeline.

    Plan:

    • Modify .github/workflows/ci.yml to run the BrowserAdapter integration suite in a separate serial test invocation
    • Keep the rest of the test suite running in parallel
    • The fixture's port binding should be made robust against parallel execution OR the suite runs serially in its own step
    • Ensure the job stays within 20-minute timeout

    Acceptance Criteria:

    1. BrowserAdapter integration suite executes in CI — no longer skipped wholesale
    2. Suite passes on three consecutive CI runs on the 2-vCPU runner
    3. Rest of test suite still runs in parallel
    4. Job stays comfortably inside timeout-minutes: 20
    5. No test is weakened, deleted, or marked skipped

    Files to touch: .github/workflows/ci.yml, possibly test config or fixture setup

    Forbidden: Simply removing SKIP_BROWSER_INTEGRATION without addressing contention, serializing entire suite, deleting/weakening tests, adding skip annotations

    Confidence: 0.9

    🤖 Posted by developerz.ai — the maintainer agent, not a human.

  4. developerz-ai commented on Sep 11, 2026

    @developerz-ai
    developerz-aiBot
    on Sep 11, 2026 – with Developerz AI
    Contributor

    Want to submit a fix? I can hand this off to the maintainer's coding agent.

    🤖 Posted by developerz.ai — the maintainer agent, not a human.

  5. ivndev001 commented on Sep 11, 2026

    @ivndev001
    ivndev001
    on Sep 11, 2026
    ContributorAuthor

    Yes — please hand this off to the coding agent.

    Your scoping is accurate and the acceptance criteria are the ones that matter. Two notes for whoever picks it up:

    The three-consecutive-green-runs bar is the point, not ceremony. This suite is flaky under parallel load, and a flaky suite passes once by luck. One green run cannot distinguish a fix from a favourable scheduling order, which is exactly how it ended up skipped rather than fixed.

    There is more headroom than the timeout-minutes: 20 suggests. That ceiling was raised in #77 while the job was timing out on the Chromium dependency install. That cause is gone — the job now completes in about 35 seconds end to end (install chromium 7s, test 13s). So a serial invocation of this suite has a large budget to spend; there is no need to trade coverage for speed here.

    Preference between the two shapes named in the plan: making the fixture's port binding robust is the better fix, because a fixture that only works when nothing else is running is the underlying fragility. A separate serial step is acceptable if the first turns out to be more than 2h of work — but say which you chose and why, rather than doing the easy one silently.

  6. developerz-ai
    added
    ready-for-agent
    on Sep 11, 2026

  7. developerz-ai commented on Sep 11, 2026

    @developerz-ai
    developerz-aiBot
    on Sep 11, 2026 – with Developerz AI
    Contributor

    Dispatched to the maintainer's coding agent (task tsk_74ccad0ea4d54ccda1e87da2a62a47c2). It will pick up from the scoped plan and begin work.

    🤖 Posted by developerz.ai — the maintainer agent, not a human.

  8. developerz-ai commented on Sep 11, 2026

    @developerz-ai
    developerz-aiBot
    on Sep 11, 2026 – with Developerz AI
    Contributor

    ci: run BrowserAdapter integration suite in CI serially

    • Picked up
    • Surveying the repo — 2026-09-11 12:22 UTC
    • Working — 2026-09-11 12:26 UTC
    • Pull request opened
    • Done

    Finished — no pull request is linked to this run.

    9m · 132 tool calls · 10 messages

    Sessions: coding run

    🤖 developerz.ai maintainer bot — sebby-ovh-3 (developerz-ai[bot])

  9. ivndev001 commented on Sep 11, 2026

    @ivndev001
    ivndev001
    on Sep 11, 2026
    ContributorAuthor

    Re-dispatching. The first attempt settled no_work on a tool defect, not on this issue's premises — and that defect is now fixed.

    Task tsk_74ccad0ea4d54ccda1e87da2a62a47c2 surveyed the checkout and reported:

    Glob `.github/workflows/*.yml`, `**/ci.yml`, `**/.github/workflows/*`, `.github/**/*`,
  `**/workflow*` — 0 paths.
`CONTRIBUTING.md:23` says CI is `.github/workflows/ci.yml` — file does not exist.
outcome: no_work, falsified: 4

    That file does exist — 5,885 bytes, present in the tree at the run's own base_sha a277712b. The coding agent's reasoning was sound throughout: it cross-checked CONTRIBUTING.md, found it pointing at a file its tools said was absent, and refused to invent work rather than hallucinate one. Its tools lied to it. code.glob and code.grep ran with dot: false, so every dot-directory was invisible.

    Fixed in developerz-ai/developerz.ai#3728 (PR #3729, squashed 43d3df34): both scanners now see dot-directories, while credential dotfiles (.env*, .npmrc, .netrc) stay excluded — a listing is reconnaissance, and grep is sharper still since it returns contents rather than paths.

    Nothing about this issue's scope has changed. The acceptance criteria stand as written, and the three-consecutive-green-runs bar in particular is the substance: this suite is flaky under parallel load, and one green run cannot distinguish a fix from a lucky scheduling order.

    @developerz-ai please pick this up again.

  10. ivndev001
    added
    ready-for-agent
    and removed
    ready-for-agent
    on Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/cienhancementNew feature or requestready-for-agentsize/2h

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions