Pre-launch validation: internal test matrix, then a staged first tranche

[htsukamoto5]

Phase 3 — the gate before the run scales up.

Revised: this issue previously called for a separate paid pilot with real strangers. That framing was wrong on the economics. Most of what it listed does not need paid participants at all, and the part that does is better obtained as the first tranche of the real study than as a throwaway pilot.

Both experiments have so far only been play-tested as two tabs on one machine, which exercises none of the failure modes that threaten a paid run.

Part 1 — Internal test matrix (free, and better than a pilot for these)

Internal testing is not a cheap substitute here. For anything adversarial or systematic it is strictly superior, because you can script the case and repeat it:

• Device / browser matrix. Systematic coverage beats whatever participants happen to own. Confirm the desktop-only block works and that the 6×2 grid is usable at realistic window sizes.
• Scripted mid-round dropout, on command and repeatable. Confirm the survivor sees the disconnect screen promptly, reaches the partner-dropped code, and that their data — including chat_transcript up to the drop — arrives at the save target (Attrition: detect partner dropout and abort the game loop with a payment path #5, Data: no save target exists — wire up DataPipe with incremental saves #3).
• Waiting room with 6–8 simultaneous testers. Room bucketing, odd-arrival handling, and ghost entries are all n>2 logic that two tabs cannot reach. Deliberately script arrival patterns: all at once, one every 30 s, and one arriving exactly as another times out. This is more thorough than real traffic, which will not reliably produce the awkward cases (Pairing: build a rolling waiting room (room bucketing, ghost expiry, odd arrivals) #10).
• Adversarial cases real participants will not reliably generate: close the tab at the exact moment of submit, refresh mid-round, open three tabs, kill the network during a flush.
• Screener branching logic and the four completion codes, end to end (Prolific plumbing: completion codes and submission redirect #7, Prolific wrapper: consent, instructions, comprehension check, debrief, mobile block #13).
• DataPipe writes, including that a partially-saved session is parseable and that the save fires on the screen-out and abort paths.
• Naive testers for trial duration — people who do not know the task, to check whether round_timeout (180 s C&WG) truncates genuine trials. C&WG trial 1 on a 12-figure board is the longest in the study and a cap that bites there would bite where the effect is largest. Treat this as a signal, not a calibration: someone doing a favour has different motivation from a paid stranger.

Part 2 — Staged first tranche (not a separate pilot)

Launch the real study capped at ~3–4 dyads, pause, inspect, then release the rest.

The economics are the whole argument. A separate pilot costs the same and produces nothing you keep; a first tranche produces the same information and those dyads count toward the target if everything worked. Marginal cost is ≈ $0 in the success case and an identical loss in the failure case, so there is no scenario where a separate pilot wins.

What only real participants can give you

• Real arrival rate. Sets the lobby timeout, which sets both the advertised duration and the no-match payment. Two of those are fixed at publish and the screen-out reward cannot be changed at all (§4). Coordinated testers cannot estimate this.
• The abort heuristic's false-positive rate. Attrition: detect partner dropout and abort the game loop with a payment path #5 infers dropout from N consecutive ended_by: "timeout" rounds, and whether N=2 or N=3 is right depends on how often real dyads legitimately time out. Testers who know the task never legitimately time out, so internal testing can confirm the mechanism fires but cannot calibrate the threshold. Set it too low and live dyads abort mid-study — destroying their data and paying two people for nothing.
• Real per-participant dropout rate, replacing the assumed 15% / 30% in §5.
• Comprehension-check failure rate and whether naive strangers actually produce the reference-shortening effect under text chat with these instructions.
• Prolific-side integration at volume: screen-out slot behaviour, approval flow, and that completion codes register as real submissions.

Inspection gate — operational metrics ONLY

The pause must look at operational metrics and explicitly not at the DV. Peeking at the outcome measure and then deciding whether to continue is exactly the flexibility pre-registration exists to remove.

Checklist, written in advance so the boundary is pre-empted rather than resisted in the moment:

• Did every recruited participant reach a completion code?
• Did rows arrive at the save target for every exit path, including screen-outs and aborts?
• Is every participant matchable by PROLIFIC_PID and every dyad by dyad_id (Data: attach dyad_id and Prolific identifiers to every row #4)?
• Observed arrival rate, lobby wait times, and no-match rate.
• Observed round-timeout rate in dyads where both partners stayed — this is the input to the Attrition: detect partner dropout and abort the game loop with a payment path #5 threshold.
• Observed session duration vs. the advertised estimate.
• Screen-out rate by screened_out_reason, and screen-out slots consumed.

Caveats worth stating

• 3–4 dyads still does not really test concurrency. Build the waiting room defensively — idempotent room assignment, tolerate duplicate entries — rather than trusting a small tranche to validate it (Pairing: build a rolling waiting room (room bucketing, ghost expiry, odd arrivals) #10).
• Scripted testers give you waiting-room logic but not the real arrival distribution. That is what Part 2 is for.

See READINESS.md sections 2, 4 and 5.