From 654aa077d9c3420b51ec3153dc86d6d417a6894f Mon Sep 17 00:00:00 2001 From: Hannah Tsukamoto Date: Wed, 29 Jul 2026 18:24:33 -0400 Subject: [PATCH 01/10] docs: add Prolific readiness audit; correct adapter-swap claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audits both reference-game builds against what a live paid Prolific run requires, verified by reading the vendored bundles rather than trusting the header comments. Headline findings: no pairing mechanism for strangers (the lobby waits unbounded, so an unmatched participant can neither exit nor be paid), no partner-dropout handling (round_timeout restarts per round, so a survivor grinds through ~71 min of dead waiting on Hawkins with no indication anything is wrong), and no data save target at all (initJsPsych takes no options; the only egress is a voluntary download button, so every non-completing participant contributes zero data). Also corrects a false claim in both HTML headers and in README: swapping adapter-multiplayer-local for -firebase is NOT a drop-in. SEED reads ?mp_session=, which the LocalAdapter constructor generates and Firebase will not, so every dyad would silently share one trial order. No behavior change — documentation and comments only. Co-Authored-By: Claude Opus 5 --- READINESS.md | 311 ++++++++++++++++++++++++++++++++++++ README.md | 23 ++- reference-game-cwg.html | 9 +- reference-game-hawkins.html | 9 +- 4 files changed, 338 insertions(+), 14 deletions(-) create mode 100644 READINESS.md diff --git a/READINESS.md b/READINESS.md new file mode 100644 index 0000000..5c840aa --- /dev/null +++ b/READINESS.md @@ -0,0 +1,311 @@ +# Prolific readiness audit + +Status of `reference-game-hawkins.html` and `reference-game-cwg.html` as candidates for a **real paid +study on Prolific**, audited 2026-07-29 against the working tree of +`chore/vendor-multiplayer-bundles` (the only branch that currently runs — `main` still points at the +broken npm `0.1.0` pins). + +Findings were verified by reading the four vendored bundles in [`vendor/`](vendor/), not just the +experiment files. Where this document contradicts a header comment in the HTML, this document is +right and the comment has been corrected. + +**Bottom line:** the experiment logic is in good shape. The gap is that there is **no pairing +mechanism, no attrition handling, and no data save target at all**. Two of those three are fixable +without waiting on upstream. + +**Decision: run `reference-game-cwg.html` as the first paid study, not Hawkins.** 6 trials (~25 min) +vs 72 trials (~55 min) changes pair-based attrition economics by roughly 3× (see +[Cost](#5-cost)), and the C&WG words-per-figure effect (41→8) is larger and more robust than +Hawkins' block-wise shortening. + +--- + +## Contents + +- [1. Pairing](#1-pairing) — blocked upstream, and bigger than "one script-tag swap" +- [2. Attrition](#2-attrition) — the survivor grinds, silently +- [3. Data](#3-data) — no save target exists +- [4. Prolific plumbing](#4-prolific-plumbing) — absent +- [5. Cost](#5-cost) +- [What's fine](#whats-fine) +- [Blocked on unpublished packages](#blocked-on-unpublished-packages) +- [Work plan](#work-plan) + +--- + +## 1. Pairing + +### The adapter-swap claim was false + +The header comments claimed that swapping `adapter-multiplayer-local` → +`adapter-multiplayer-firebase` required no other change to the file. At least three things change: + +1. **`SEED` breaks silently — highest-severity item in this section.** `SEED` reads `?mp_session=` + (`reference-game-hawkins.html`, `reference-game-cwg.html`), and that query param is *generated by + the LocalAdapter constructor* — confirmed in `vendor/adapter-multiplayer-local.js`, which writes + it via `history.replaceState`. Firebase will not produce it. Every dyad therefore falls through to + the literal `"cwg-fullboard"` / `"hawkins-sequential"` fallback, so **every dyad receives an + identical trial order**, confounding item order with trial number — precisely the confound the + seeding comment says the design exists to prevent. This fails with no error and produces + valid-looking, unusable data. +2. **`persistParticipant: true`** is a LocalAdapter-only constructor option. +3. `MIN_PLAYERS` gating assumes a private two-person session. Firebase provides a *shared room*, + which is a different problem — see ghost pairing below. + +### There is no waiting room + +`lobbyTrial` (`jsPsychMultiplayerSync`) is configured with **no `timeout`**. The plugin's default is +`null`, meaning `wait()` is called unbounded. A participant who arrives and is never matched **waits +on that screen forever**: no timeout, no `on_timeout`, no exit, no completion code, no payment path. + +Unmatched participants **must still be paid** — they gave up their time. Right now they cannot even +submit. + +Note that a rolling waiting room is **not** something the Firebase adapter provides. `sync` and +`role` are per-session barriers, not a matcher. Bucketing arrivals into two-person rooms, expiring +stale rooms, and handling the odd arrival is logic that has to be written on top, and it is the +largest single piece of missing work in this audit. + +### Ghost pairing + +`wait_for` counts any group entry carrying a `name`. The adapter only removes a participant's entry +on an **explicit `disconnect()`**. There is no heartbeat and no presence tracking anywhere in the +four bundles (`presence`, `heartbeat`, and `beforeunload` each return zero matches). A tab closed in +the lobby leaves its entry behind. + +With persistent Firebase room state, a live participant will be "matched" with a ghost, assigned a +role, and then time out on every single round. Silent, and frequent. + +### Spectator dead-end + +The `spectator` overflow path shows "This game is already full" for four seconds, disconnects, and +ends **with no completion code**. Anyone who is the odd person out is unpaid and unable to submit. + +--- + +## 2. Attrition + +The survivor does not hang. **They grind**, which is worse, because it consumes their time before +they give up. + +`round_timeout` sets its timer at **round start**. When it fires the round ends with +`ended_by: "timeout"` and a null assignment, feedback shows, and the timeline **advances to the next +round**, starting a fresh timer. There is no cross-round abort and no presence check: + +| | per-round dead wait | rounds left if partner drops on round 1 | total dead time | +| --- | --- | --- | --- | +| **Hawkins** | 60 s | 71 | **~71 min** | +| **C&WG** | 180 s | 5 | **~15 min** | + +And the survivor is **never told anything**. The plugin contains no partner-status copy, no +"disconnected" state, no UI affordance — searching for waiting/partner-status strings returns +nothing. They see a normal, fully interactive board that simply never responds, 72 times over. + +Realistically they abandon within minutes, so **the survivor is lost too**, and their data never +leaves the browser (§3). One dropout burns two payments *and* generates a support ticket. + +### Missing, in priority order + +- [ ] A **presence/heartbeat signal** so a client can know the partner is gone. *(needs adapter + support — see [blocked](#blocked-on-unpublished-packages))* +- [ ] A **"your partner disconnected" screen that breaks out of the game loop.** + `conditional_function` on `gameLoop` is evaluated once, so a mid-game abort needs either a + per-round `conditional_function` on `gameRound` or `jsPsych.abortCurrentTimeline()` from the + round's `on_finish` when `ended_by === "timeout"`. +- [ ] A **survivor payment path**: partial completion code, data flushed, honest debrief. Highest + value item in this section and **not blocked upstream** — two or three consecutive + `ended_by: "timeout"` rounds is a perfectly good dropout proxy, no presence API required. +- [ ] A decision on **partial-data usability**. A dyad that dies at C&WG trial 4 still has 3 clean + trials; for a 6-trial design that is worth keeping. Requires `n_trials_completed` in the data + to filter on. **This decision determines what the abort path does, so make it first.** + +--- + +## 3. Data + +### There is no save target. Cheapest catastrophic bug here. + +`initJsPsych()` is called with **no options at all** — no `on_finish`, no `on_data_update`. The only +egress in either file is a `localSave` button on `doneTrial` that the participant must voluntarily +click. No DataPipe, no OSF, no endpoint, no `fetch` anywhere in the repo. + +Consequences: + +- **Every participant who does not reach the final screen contributes zero data.** All dropouts, all + survivors who bail, all lobby timeouts, all spectators — precisely the sessions needed to + characterize attrition. +- Even completers only contribute if they click a button labelled "Download data (JSON)" on a screen + that already says "You may close this tab." Most will not. +- You are paying for data you will not receive. + +**Fix (unblocked, do first):** jsPsych DataPipe (`pipe.jspsych.org` → OSF) needs roughly five lines. +Wire it to **incremental** saving, not only `on_finish` — with pair-based attrition, `on_finish`-only +saving loses exactly the sessions that matter. Save per-round, or on the abort path. + +### Content gaps once egress exists + +- [ ] **No `jsPsych.data.addProperties()` anywhere**, so no `PROLIFIC_PID`, `STUDY_ID`, or + `SESSION_ID` is attached to any row. Without these you cannot reconcile data against Prolific + submissions or approve payments. +- [ ] **No dyad identifier on data rows.** Subtle and important: the entire analysis is dyad-level + (shortening *within* a dyad across blocks), and nothing on a row currently says which two + participants formed the pair. `save_group: true` on `roleTrial` gives one group snapshot; + `partner_id` is in scope but never written to data. A stable `dyad_id` (session/room id) + belongs on every row. +- [ ] No `BLOCKS` / `TRIALS` / `round_timeout` config recorded, needed for provenance when piloting + at different values. +- [ ] No screen/browser/viewport capture, and **no mobile block** — a 6×2 tangram grid on a phone is + a different task. + +--- + +## 4. Prolific plumbing + +Essentially 100% absent. The only repo-wide matches for `prolific|consent|debrief|comprehension` are +prose mentions in `README.md` and the HTML header comments. + +| Required | Status | +| --- | --- | +| PID / STUDY_ID / SESSION_ID capture from URL | **missing** | +| Completion-code redirect (`app.prolific.com/submissions/complete?cc=…`) | **missing** — no participant can submit at all | +| Separate codes for complete / partner-dropped / no-match | **missing** (all three are needed) | +| Consent form | **missing** | +| Instructions | **missing** — the only task explanation is the one-line `prompt` on the game screen | +| Comprehension check | **missing** | +| Debrief | **missing** | +| Fullscreen / mobile block | **missing** | +| Timing estimate shown to participant | **missing** | + +Two items beyond the standard list that matter specifically for a *dyadic* study: + +- [ ] **Instructions must set partner expectations** — that a real person is waiting, and that + abandoning costs someone else their payment. In dyadic work this framing measurably reduces + mid-task dropout, and it is free. +- [ ] **Instructions and the comprehension check must run BEFORE the lobby.** Currently + `nameTrial` → `lobbyTrial` leaves one participant reading nothing while the other waits. + Anything slow goes before pairing. The files already apply this reasoning correctly to + `preloadTrial`, so extend it. + +Minor: + +- [ ] `nameTrial` collects a free-text display name — participant-entered PII shown to a stranger. + Assign neutral labels ("Partner A") or filter; cheaper than an IRB conversation. +- [ ] `jsPsychPreload` defaults to halting on error. One 404 in `assets/tangrams/` leaves a paid + participant dead in the water with no completion code. Set `on_error` / + `continue_after_error` deliberately. + +--- + +## 5. Cost + +Assuming Prolific's £9.00/hr recommended rate and a 33% service fee on participant pay. + +**C&WG** (~25 min: instructions + comprehension + lobby + 6 trials) → £3.75/participant, +**£4.99 all-in**. Attrition is multiplicative on dyads; at ~15% per-participant mid-task dropout +(reasonable for a 25-min synchronous task with good framing), dyad survival ≈ 0.72: + +| | for 20 usable dyads | +| --- | --- | +| dyads to start | ~28 | +| participants recruited | ~56 | +| of which usable | 40 | +| **paid but unusable** | **~16** (dropouts + their stranded survivors) | +| **cost** | **~£280** | + +Add ~10–15% for unmatched-in-lobby participants who must be paid prorated, plus headroom for one +wasted session while learning real arrival-rate and attrition numbers. **Budget ~£350 for the first +real run.** + +**Hawkins** (~55 min) → £8.25/participant, **£10.97 all-in**, and 72 synchronous trials realistically +push per-participant dropout to ~30%, so dyad survival ≈ 0.49 → ~41 dyads → ~82 participants → +**~£900** for the same 20 dyads. Three-plus times the cost, with the §2 survivor-grind failure mode +at its worst. + +### Two structural cost levers, both large + +- **Simultaneous arrival is the real constraint.** Pair-based studies need participants + *concurrently*, which Prolific does not guarantee. You control it by launching in a tight burst + rather than leaving the study open — which is why the waiting room and its timeout policy are + load-bearing on cost, not just on UX. +- **Partial-data salvage.** With C&WG's 6 trials, a dyad dying at trial 4 yields 3 usable trials. If + the analysis can use partial dyads, effective attrition cost drops substantially. + +--- + +## What's fine + +Worth stating explicitly, because the list above is long: + +- **The replication logic itself.** Both files were verified parameter-by-parameter against the + papers and against `hawkrobe/tangrams`, and the header comments documenting those decisions are + accurate and unusually thorough. +- **The per-round data spec** covers every stated DV: `message_count`, `messages_sent`, + `chat_transcript`, `accuracy`, `n_correct`, `correct`, `rt`, `my_order` / `partner_order`, + `ended_by`, plus `interaction_history` (enabled in both files). Analysis-side, once data reaches + you, you are in good shape. +- **`ended_by: "timeout"` + null assignment** behaves exactly as the header comments describe; + verified in the plugin's finish path. The documented analysis filter works. +- **`chat_transcript` is built from `getAll()`** — the *full* shared transcript, not just the local + client's messages. A survivor's saved rows therefore contain the complete dialogue up to the drop, + including the departed partner's messages. **The core DV survives a dropout**, provided the data + can get out (§3). +- **Preload runs before pairing**, correctly, so nobody waits on someone else's image fetches. +- **`vendor/` pinning is correct and internally consistent.** Bundles built from `69c0d7b`, loaded + from commit `5046bf0`, and both HTML files pin `5046bf0`. The stale-pin hazard `vendor/README.md` + warns about is real but not currently triggered. + +> **Do not "fix" the `vendor/` pins back to npm.** The published `0.1.0` builds predate the +> `jsPsych.multiplayer` namespace migration and do not work with the pinned core. `main` currently +> has exactly this bug, having reverted the vendoring. See [`vendor/README.md`](vendor/README.md). + +--- + +## Blocked on unpublished packages + +Tracked separately because these cannot be unblocked from this repo. + +1. **`adapter-multiplayer-firebase` is unpublished** → no cross-device pairing at all. Everything in + §1 and §2 is untestable end-to-end until this lands. **This is the hard gate on a paid run.** +2. **No presence/disconnect API in any bundle**, confirmed by inspecting all four. The proper fix for + §2 — *knowing* the partner is gone rather than inferring it from timeouts — needs either the + Firebase adapter exposing presence (Firebase RTDB has `onDisconnect`, so it plausibly does or + could) or an upstream plugin change. **Verify this specifically when the adapter is available**; + it is the difference between a good survivor experience and a tolerable one. The timeout-counter + workaround in §2 does not depend on it. +3. **`plugin-multiplayer-reference-game` is unpublished**, and the other three have only + pre-namespace-migration `0.1.0` releases. Gated on + [jspsych-multiplayer PR #35](https://github.com/jspsych/jspsych-multiplayer/pull/35). + +--- + +## Work plan + +Everything in phase 1 is unblocked, and it is most of the total effort. + +### Phase 1 — now, no upstream dependency + +- [ ] 1. **Data egress** (DataPipe/OSF, incremental saves). Highest value per line of code in this + document; without it every other fix produces nothing analyzable. +- [ ] 2. **Dyad + Prolific identifiers** on every row via `addProperties` (`dyad_id`, + `PROLIFIC_PID`, `STUDY_ID`, `SESSION_ID`). +- [ ] 3. **Timeout-based dropout detection + abort path** (N consecutive `ended_by: "timeout"` → + flush data → "partner disconnected" screen → partial completion code). Kills the 71-minute + grind with no presence API. +- [ ] 4. **Lobby timeout + no-match exit + payment path**, and fix the spectator dead-end. +- [ ] 5. **Prolific wrapper**: consent, instructions with partner-expectation framing, comprehension + check, debrief, three completion codes, mobile block. All before the lobby. +- [ ] 6. **Decide the partial-data policy** — it changes what #3 does, so settle it early. + +### Phase 2 — after `adapter-multiplayer-firebase` publishes + +- [ ] 7. **Fix `SEED` derivation for Firebase** (§1). Do this *in the same change* as the adapter + swap, or you will collect a wave of confounded data that looks fine. +- [ ] 8. Build the **rolling waiting room** (room bucketing, stale-room expiry, odd-arrival + handling). Largest remaining piece. +- [ ] 9. **Verify whether presence is exposed**; if so, upgrade #3 from timeout-inference to real + disconnect detection. + +### Phase 3 + +- [ ] 10. **Two-device pilot with real strangers**, deliberately including a scripted mid-round + dropout, before spending money. diff --git a/README.md b/README.md index 2407807..af7e13e 100644 --- a/README.md +++ b/README.md @@ -45,13 +45,20 @@ These currently run on `adapter-multiplayer-local` (no backend needed) for two-t into a second tab so a second player joins. 4. The first tab becomes the director, the second the matcher. -For a paid Prolific run, swap `adapter-multiplayer-local` for -[`adapter-multiplayer-firebase`](https://github.com/jspsych/jspsych-multiplayer/tree/main/packages/adapter-multiplayer-firebase) -(one script-tag swap, see the header comment in each file) plus a real waiting room for pairing. +## Running it as a paid study -**Note on package versions:** the `@jspsych-multiplayer/*` package script tags below are pinned to -`0.1.0` on jsDelivr, but those packages are not yet published to npm — publishing is gated on +**Not yet possible — see [`READINESS.md`](READINESS.md)** for a full audit. In short: there is no +pairing mechanism for strangers, no partner-dropout handling, and no data save target, and +cross-device pairing is gated on the unpublished +[`adapter-multiplayer-firebase`](https://github.com/jspsych/jspsych-multiplayer/tree/main/packages/adapter-multiplayer-firebase). + +Swapping the adapter is **not** a one-script-tag change: `SEED` is derived from the `?mp_session=` +param that the *local* adapter generates, so on Firebase every dyad would silently receive the same +trial order. `READINESS.md` lists what actually has to change, and which items are blocked upstream. + +**Note on package versions:** the four `@jspsych-multiplayer/*` bundles load from +[`vendor/`](vendor/README.md) in this repo, not npm — `plugin-multiplayer-reference-game` is +unpublished and the other three have only pre-namespace-migration `0.1.0` releases, which do not work +with the pinned core. Publishing is gated on [jspsych-multiplayer PR #35](https://github.com/jspsych/jspsych-multiplayer/pull/35) ("Version -Packages") merging. Until then, either build the packages from a local checkout of -jspsych-multiplayer and swap in relative `dist/` paths, or wait for the publish and confirm the -pinned version still matches. +Packages"). Do not repoint these to npm until that merges. diff --git a/reference-game-cwg.html b/reference-game-cwg.html index ef884c5..25f22da 100644 --- a/reference-game-cwg.html +++ b/reference-game-cwg.html @@ -8,9 +8,12 @@ - + +