[#4758 P1] perf-runner driver crash: setInterval race condition closes Playwright context mid-scrape

[OneStepAt4time]

Summary

The perf-runner drivers (driver.mjs, driver-final.mjs) crash with page.evaluate: Target page, context or browser has been closed due to a race condition between concurrent setInterval callbacks sharing the same Playwright page object. This is the root cause of #4758 (watch=not-yet-launched for 48h+).

Fix is already in place at /home/buntu/.aegis-perf-run/driver-final.mjs and driver.mjs (in-place edit, no PR — drivers live outside the aegis repo). Regression test at /home/buntu/.aegis-perf-run/driver-race-test.mjs (2/2 passing).

Symptom

• Driver starts, runs cycles 1-5 (or 1-11 in phase 1) successfully.
• After ~30-50 minutes, a cycle tick fires the route-navigation block, which takes longer than SCRAPE_MS (30s).
• The next setInterval fires while the cycle is still in progress.
• Both callbacks touch the same page object: one is doing page.goto, the other is doing page.evaluate.
• Playwright throws page.evaluate: Target page, context or browser has been closed.
• 5 consecutive failures, then the process dies with no clean shutdown.

Evidence

{"t":"2026-06-15T14:28:20.613Z","level":"error","msg":"scrape failed","label":"scrape:94","message":"page.evaluate: Target page, context or browser has been closed"}   <- driver.mjs (Phase 1)
{"t":"2026-06-15T15:21:10.341Z","level":"info","msg":"cycle tick","cycleCount":6,"elapsed":1501723}                                        <- driver-final.mjs (Phase 2)
{"t":"2026-06-15T15:22:10.359Z","level":"error","msg":"scrape failed","label":"scrape:52","message":"page.evaluate: Target page, context or browser has been closed"}
{"t":"2026-06-15T15:22:40.380Z","level":"error","msg":"scrape failed","label":"scrape:53","message":"page.evaluate: Target page, context or browser has been closed"}
{"t":"2026-06-15T15:23:10.408Z","level":"error","msg":"scrape failed","label":"scrape:54","message":"page.evaluate: Target page, context or browser has been closed"}
{"t":"2026-06-15T15:23:40.437Z","level":"error","msg":"scrape failed","label":"scrape:55","message":"page.evaluate: Target page, context or browser has been closed"}
{"t":"2026-06-15T15:24:10.444Z","level":"error","msg":"scrape failed","label":"scrape:56","message":"page.evaluate: Target page, context or browser has been closed"}
[log ends — no clean shutdown, process dead at 15:24]

The 30s spacing between failures matches SCRAPE_MS=30000. The cycle tick at 15:21:10 ran the 9-route block; the next setInterval at 15:21:40 (30s later) raced against the still-in-progress cycle, and the page was being torn down by page.goto when the new page.evaluate fired.

Root cause

driver-final.mjs:73-87 (pre-fix):

const timer = setInterval(async () => {
  const elapsed = Date.now() - startedAt;
  if (elapsed >= totalMs) { ...; process.exit(0); }
  scrapeCount++;
  await scrape(`scrape:${scrapeCount}`);
  if (elapsed % cycleMs < scrapeMs + 1000) {
    cycleCount++;
    log('info', 'cycle tick', { cycleCount, elapsed });
    for (const route of ROUTES) {
      try { await page.goto(`${AEGIS_BASE}/dashboard${route}?perf=1`, ...); await scrape(`cycle:${route}`); }
      catch (e) { ... }
    }
    try { await page.goto(DASH_URL, ...); } catch (e) { ... }
  }
}, scrapeMs);

The cycle block does 9 route navigations (each with its own page.goto + scrape) + 1 return navigation. With ~3-5s per route, the cycle takes 30-50s — longer than scrapeMs (30s). setInterval does not wait for the previous callback to complete, so the next callback fires while the cycle is still in progress, racing on the shared page object.

driver.mjs (Phase 1) has a different but related form: two separate setIntervals (scrapeTimer 30s, cycleTimer 5min) — quickRouteCycle can run 30s+ for 9 routes, racing against the next scrapeTimer fire.

Fix

In-place edit to both driver-final.mjs and driver.mjs. Add an inFlight flag that guards both timers' callbacks. Second concurrent tick is skipped with a warn log.

let inFlight = false;
const timer = setInterval(async () => {
  if (inFlight) { log('warn', 'tick skipped: previous tick still in flight'); return; }
  inFlight = true;
  try {
    // ... existing tick logic ...
  } catch (e) {
    log('error', 'tick failed', { message: e.message });
  } finally {
    inFlight = false;
  }
}, scrapeMs);

The try/catch/finally ensures inFlight is always reset, even on thrown errors. The warn log on skipped ticks provides observability into how often the race is being prevented (useful telemetry for sizing SCRAPE_MS vs route-load time).

Regression test

/home/buntu/.aegis-perf-run/driver-race-test.mjs (Node --test):

$ node --test /home/buntu/.aegis-perf-run/driver-race-test.mjs
ok 1 - buggy setInterval pattern: concurrent ticks produce page errors
ok 2 - fixed inFlight pattern: concurrent ticks are skipped, no page errors
# tests 2 / pass 2 / fail 0

Test 1 reproduces the bug (5 concurrent ticks at 30ms intervals → page errors). Test 2 verifies the fix (5 concurrent ticks with inFlight guard → 0 errors, 2 skipped).

Verification

• node --test on the regression test: 2/2 pass.
• Code review of the fix: inFlight is set before any work, reset in finally, and the second concurrent tick is skipped with a warn log. The try/catch/finally block ensures the flag is never stuck true.
• The drivers themselves are not run end-to-end here (would require a running aegis dashboard + 30+ minute endurance run); the unit test exercises the same race-condition pattern with a mock page.

Out of scope (deferred)

• Refactor: extract the tick logic into a shared module and import from the driver. Currently the fix is duplicated in both driver.mjs and driver-final.mjs. Acceptable for now; if a third driver variant appears, the refactor should land.
• Move perf-runner into the aegis repo as a tracked tool (tools/perf-runner/ or similar). Currently the drivers live in /home/buntu/.aegis-perf-run/ outside any git repo. Out of scope for [Endurance #4683] PHASE2-WATCH reports watch=not-yet-launched — no endurance driver running since 2026-06-15 17:24 #4758; separate ticket.
• The wider Athena triage outcome ("Hephaestus driver-bug") included "file P1 with repro" — this issue is the repro+fix. Re-arm gate is held by P0: isolated agentTurn sessions time out on all 5 LLM providers (release-please + dogfooding blocker) #4755 close + [P1] Endurance test: 4h+ CC development session through Aegis #4683 close (per Athena's update on 2026-06-17 20:47 GMT+2).

Lane

• Hephaestus (implementation, this fix)
• Re-arm of the perf-runner: blocked on P0: isolated agentTurn sessions time out on all 5 LLM providers (release-please + dogfooding blocker) #4755 (P0 LLM upstream) + [P1] Endurance test: 4h+ CC development session through Aegis #4683 (P1 endurance test) close per Athena.

Refs

• [Endurance #4683] PHASE2-WATCH reports watch=not-yet-launched — no endurance driver running since 2026-06-15 17:24 #4758 (the parent: watch=not-yet-launched for 48h)
• [P1] Endurance test: 4h+ CC development session through Aegis #4683 (endurance test, In Progress/P1)
• P0: isolated agentTurn sessions time out on all 5 LLM providers (release-please + dogfooding blocker) #4755 (LLM upstream P0, blocks re-arm)
• Hermes's diagnostic on 2026-06-17 20:49 GMT+2 ("Driver is dead, not running... Crash signature matches 'Hephaestus driver-bug'")
• Athena's triage decision tree on 2026-06-17 20:47 GMT+2 (Branch B = Hephaestus driver-bug lane)

[OneStepAt4time]

Closing as the fix is verified in place and the regression test passes (re-ran just now: node --test driver-race-test.mjs → 2/2 pass, ~417ms).

Verification evidence:

• driver-final.mjs L77-79, 98: inFlight guard with try/catch/finally and warn log on skipped ticks
• driver.mjs L168-178, 182-192: shared inFlight flag guarding both timers (scrapeTimer + cycleTimer)
• Regression test: node --test driver-race-test.mjs → 2/2 pass
  • Test 1 reproduces the race (5 concurrent ticks at 30ms → page errors with the buggy pattern)
  • Test 2 verifies the fix (5 concurrent ticks with inFlight guard → 0 errors, 2 skipped)

What this close does NOT change: the parent #4758 (watch=not-yet-launched) remains open because the re-arm of the perf-runner is still gated on #4755 (P0 LLM upstream) + #4683 (P1 endurance test) close, per Athena's 2026-06-17 20:47 GMT+2 update. When those close, the driver can re-launch and the watch will come back online.

Why no PR: drivers live in /home/buntu/.aegis-perf-run/ outside the aegis repo, so the issue is the audit trail (as designed).

— Hephaestus (verification run 2026-06-17 21:21 GMT+2)