Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions apps/server/api/docs/testing-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,11 @@ variable and gets no net, correctly, since it also gets the shared default
socket dir, where killing anything would reach panes this run never started.

**The net covers this package only.** `packages/pane-runtime` and
`apps/node/agent` also spawn real tmux and run a bare `bun test`, so nothing
sweeps behind them; what stands there is each suite's own `afterAll`, which
registers a socket BEFORE spawning on it (`freshSocket` in
`tmux-runner.test.ts`) and so has no window to leak through. That is a
`apps/node/agent` also spawn real tmux, so what stands there is each suite's
own `afterAll`, which registers a socket BEFORE spawning on it (`freshSocket`
in `packages/pane-runtime/src/__tests__/helpers/tmux-test-harness.ts`) and so
has no window to leak through; under `--parallel` the helper is a per-file
module, which is exactly per-worker. That is a
narrower guarantee than this one (it holds as long as every future suite
keeps registering first), and it is stated here rather than fixed because
extending the variable to those packages is a change to how their sockets are
Expand Down
28 changes: 28 additions & 0 deletions docs/superpowers/specs/2026-09-29-parallel-tests-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,34 @@ after PR 1; split another job only if a real long pole remains.
48 cores keep 12/4), and the daemon test now waits for both unlinks.
No package fell back to `parallel = 1`.

## Round 2 (post-#277, issue #261 continued)

The flag landed (#278, merged 2026-09-30) with CI serial + `--retry=1` + a
2-way `--shard` matrix; the server+node job's CI floor halved (4 m 36 s serial
single-job → 2 m 28 s for the longer shard). The remaining CI reds turned out
to be five distinct load-sensitive tests, each a real flake the sharding made
visible, fixed on that PR: `pane-repaint` ENOENT (missing capture-child log now
reads as zero bytes, the `paneReadsAsBooting` equation), the desktop action
runner's re-probe landing in the NEXT test's fake (the harness now retires the
query clients on `restore()`), `daemon.test.ts` peeking `firstFrames` off an
`opens` gate, and two remote-attach/repaint waits that gated on one async fact
and consumed a sibling. `apps/server/web` gained `--timeout 30000` (its act
chains measured 26 s on the throttled container); the two `NetworkPluginCard`
per-test 20 s ceilings came off to that package knob. CI reached full green
on the third desktop attempt - the reset-dialog test is a known intermittent
load flake (the fake's `node_settings` always returns a plane, so a reset's
re-fetch resurrects the Control Plane screen), tracked as its own follow-up.

Two suite splits followed, both the same mechanism (under `--parallel` a
single file is a single worker, so the longest file sets the package floor):
`subshell-ws-local-attach.test.ts` (26.1 s, 41 tests) into three files over a
shared harness (#279), and `tmux-runner.test.ts` (13.8 s, 49 tests) into
stub-argv / real-input / liveness files (#280), which took the pane-runtime
package from 13.9 s to 5.3 s at width 12. Both preserve every test verbatim
(reviewer-verified byte-identical bodies); both read the launcher recorder
through ESM live bindings + a `bumpCaptureCalls()` because import assignment
is illegal.

## Non-goals

`--no-isolate`, Playwright e2e, `test:cli`, cargo tests, turbo config
Expand Down
109 changes: 109 additions & 0 deletions packages/pane-runtime/src/__tests__/helpers/tmux-test-harness.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
import { afterAll } from "bun:test";
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { spawnSync } from "bun";
import { TmuxRunner } from "../../tmux-runner.js";

/**
* Shared fixture for the three files this suite was split into (issue #261
* round 2): under `--parallel` one file is one worker, so the single 1195-line
* file set the package's wall-clock floor. Every module here is per-file under
* isolation, which is exactly right: each split file reaps its own sockets.
*/
export const runner = new TmuxRunner();

/**
* Sockets of every server these files spawn. Tests historically tracked only the
* most recent socket and killed only the SUBSHELL — a mid-test assertion failure
* skipped the kill, and a pane blocked on `read` outlived the suite (the server
* never exits while a subshell lives). `afterAll` now reaps WHOLE servers for
* every socket spawned here, so a failing test cannot leak a daemon.
*/
const spawnedSockets = new Set<string>();

/** Per-file counter behind `freshSocket`'s uniqueness. Exported as a live
* binding (read, never assigned) for the one test that names a temp file off
* the same counter. */
export let socketSeq = 0;

/**
* Returns a fresh unique socket name, already registered for the afterAll
* reaper. mktemp `-u`-style uniqueness (pid + ms + per-file counter): two
* suites running in parallel — or two `freshSocket` calls inside one
* millisecond — can never collide on a live server.
*/
export function freshSocket(kind: string): string {
socketSeq += 1;
const socket = `subshell-test-${kind}-${process.pid}-${Date.now()}-${socketSeq}`;
spawnedSockets.add(socket);
return socket;
}

/**
* Polls until `file` contains `needle`, returning its contents.
*
* Used instead of a fixed sleep for assertions that wait on tmux's
* `pipe-pane` flushing through `cat >> file`: there is no interval worth
* hardcoding, and on failure this reports what the file actually held.
*/
export async function waitForFileToContain(file: string, needle: string, timeoutMs: number): Promise<string> {
const deadline = Date.now() + timeoutMs;
let last = "";
while (Date.now() < deadline) {
last = (await Bun.file(file).exists()) ? await Bun.file(file).text() : "";
if (last.includes(needle)) return last;
await Bun.sleep(25);
}
throw new Error(
`timed out after ${timeoutMs}ms waiting for ${JSON.stringify(needle)} in ${file}; saw ${JSON.stringify(last)}`,
);
}

/**
* Writes an executable stub in a temp dir and returns both paths. The
* caller `rmSync`s the directory; nothing real is spawned.
*/
export function writeStub(script: string): { dir: string; path: string } {
const dir = mkdtempSync(join(tmpdir(), "subshell-tmux-async-"));
const path = join(dir, "tmux-stub");
writeFileSync(path, script, { mode: 0o755 });
return { dir, path };
}

/**
* Issue #244 frames: [typed payload, argv element the fix must spawn]. Only
* payloads ending in `;` change; the rest pass through byte-identical. The
* `a\;` and `a\\;` rows are where the rule is load-bearing: the parser's
* restore eats exactly ONE preceding backslash, so the encoder adds exactly
* one — enough to shield the payload's own backslashes, never one too many.
* Consumed by both the stub-argv pin and the real-tmux delivery test.
*/
export const semicolonFrames: Array<{ payload: string; encoded: string }> = [
{ payload: ";", encoded: "\\;" },
{ payload: ";;", encoded: ";\\;" },
{ payload: "x;", encoded: "x\\;" },
{ payload: "a\\;", encoded: "a\\\\;" },
{ payload: "a\\\\;", encoded: "a\\\\\\;" },
{ payload: "a;b", encoded: "a;b" },
{ payload: ";x", encoded: ";x" },
{ payload: "y;;z", encoded: "y;;z" },
];

afterAll(async () => {
// kill-server (not kill-session) on every socket this file touched: tears
// the daemon down even when a failed assertion skipped the per-test kill.
// Already-dead sockets error out; that is expected and ignored.
//
// Then UNLINK, because killing the server does not remove its socket file —
// that is the same leak `cleanSocket` exists to close in production, and
// these suites have no throwaway `TMUX_TMPDIR` the way the server's does, so
// their sockets land in the developer's real tmux dir and stayed there. A few
// hundred had accumulated by 2026-09-15.
const runner = new TmuxRunner();
for (const socket of spawnedSockets) {
spawnSync(["tmux", "-L", socket, "kill-server"], { stdout: "ignore", stderr: "ignore" });
await runner.cleanSocket(socket);
}
spawnedSockets.clear();
});
Loading
Loading