diff --git a/DECISIONS.md b/DECISIONS.md new file mode 100644 index 0000000..1e57409 --- /dev/null +++ b/DECISIONS.md @@ -0,0 +1,105 @@ +# Decision register + +Every open decision for the first paid Prolific run, with a suggested answer, why, and what it costs +to change your mind later. **All of these are suggestions, not settled calls** — the point of this +file is to make them reviewable in one pass rather than scattered across +[`READINESS.md`](READINESS.md) and issues #3–#13. + +Grouped by **when the decision stops being deferrable**, because that is what determines what needs +attention first. Reversal cost is the column to read if you are short on time: cheap ones can be +changed after launch, expensive ones cannot. + +--- + +## A. Before writing code + +| # | Decision | Suggested | Why | Reversal cost | +| --- | --- | --- | --- | --- | +| A1 | Which experiment runs first | **C&WG**, not Hawkins | ~25 min vs ~55 min; ~$465 vs ~$1,500 for 20 dyads, and it bounds the survivor-grind failure at 15 min instead of 71 (§5) | Low — Hawkins is a separate file, unchanged | +| A2 | Where phase-1 work lands | **`reference-game-cwg.html` only** | The two files are near-duplicates; back-porting before the pilot means writing every fix twice against assumptions the pilot may invalidate | Low, but rework grows the longer both drift | +| A3 | Does the abort path flush data | **Yes, unconditionally** | The only irreversible bit. Unflushed rows are gone forever; flushed rows can always be ignored. #12 needs them to characterise attrition regardless | **High** — data not saved cannot be recovered | +| A4 | DataPipe save shape | **Per-round chunks**, nonce-suffixed filenames | OSF 409 on a duplicate filename is the one failure path that is *not* retried, so a collision loses that round permanently (§3, #3) | Medium — changes analysis-side reassembly | +| A5 | Screening mechanism | **Prolific's built-in custom screening**, not a two-study design | Fits an external link with branching logic; one funnel instead of two; screen-outs auto-approve and pay | Medium — switching means re-running recruitment | +| A6 | Timeline ordering | consent → short description → exposure item → commitment gate → full instructions → comprehension → lobby | Consent must precede storing screening responses; everything pre-gate counts as paid screening time; instructions must precede the lobby so nobody waits while their partner reads (#13) | Low | +| A7 | Screening exits and codes | **One shared `screened-out` code**, separated in data by `screened_out_reason` | Prolific configures only one screen-out code per study. The two rates answer different questions and must stay separable (#7) | Low if the field exists from the start; high if retrofitted after data collection | + +## B. Locked at publish — cannot be changed afterwards + +**These are the expensive ones.** Each is fixed when the study goes live. + +| # | Decision | Suggested | Why | Reversal cost | +| --- | --- | --- | --- | --- | +| B1 | Participant pay rate | **$15.00/hr** ($6.25 per 25-min session, $8.31 all-in) | Above Prolific's $12.00/hr recommendation. Sustained real-time coordination, no pausing, and attention *is* the measurement. Partly self-funding: better pay → lower dropout → fewer dyads needed, since attrition is multiplicative (§5) | **Cannot change** for a published study | +| B2 | No-match payment | **Full task rate** (~$1.25 for a 5-min wait), not the $0.14/min floor | A floor-rate payment loses to returning the study and taking a short survey, which teaches people to abandon the lobby exactly when we need them to stay. Costs ~$10–15 across a whole run (§4) | **Cannot change** | +| B3 | Screen-out reward | **~$0.14/min of screener length**, measured in the pilot, **rounded up** | Prolific's $0.14 minimum is benchmarked to a one-minute screener, so ~$0.28 at two minutes. Above-norm rewards attract screen-out farmers, who also consume slots | **Cannot change** — round up | +| B4 | `limitSessions` | **Off** | DataPipe's counter increments per *save call*, not per participant, so per-round saving uses 7 sessions per C&WG participant. Sizing it to a participant count caps the run a seventh of the way through and drops every later save silently (§3) | Medium — but the failure it prevents is silent and mid-run | +| B5 | Advertised study duration | Must **include expected lobby wait** | Prolific policy, and it is what the lobby timeout actually sets (§4) | **Cannot change** | +| B6 | Lobby timeout | **~5 min** as a starting point, refined from observed arrival rate | Sets both B2 and B5, so it is not merely a UX number (#6) | **Cannot change** once advertised | +| B7 | Prescreeners | Desktop-only; exclude previous participants from **all** prior runs | A 6×2 grid on a phone is a different task. Tangram tasks have large practice effects, so a repeat participant is contaminated invisibly — and the pilot poisons the pool for the real run if this is not set from the very first launch | **Cannot change** | +| B8 | Screen-out slot limit | Set with headroom against expected screen-out rate | Hitting it **auto-pauses the study** — worst case mid-burst-launch, when simultaneous arrival is the whole ballgame | Medium | + +## C. Before the first paid participant — pre-registration + +Locked by the pre-registration, not by the code. See §7. + +| # | Decision | Suggested | Why | Reversal cost | +| --- | --- | --- | --- | --- | +| C1 | Analysis form | **Mixed-effects, trial as a continuous predictor**, partial dyads included and weighted | The claim is about a curve, not a two-point contrast. Partial dyads then contribute naturally with no exclusion rule | **Cannot change** after pre-registration without disclosure | +| C2 | Minimum usable partial dyad | **2 completed trials** (`ended_by: "submit"` only) | One trial informs only the intercept and contributes nothing to a slope | Same | +| C3 | Robustness check | Completers-only fit, **mandatory not optional** | Dropout is almost certainly not ignorable: dyads that break are plausibly those struggling, i.e. biased toward the high end of the DV | Same | +| C4 | Recruitment target | Power for **completers-only** (the conservative target) | Surplus power is harmless; a shortfall means topping up after seeing data, which is optional stopping | **High** — cannot be fixed after launch without breaking the pre-registration | +| C5 | Sample-size justification | Either a power analysis or an explicit statement that 20 dyads is **resource-constrained** | The 20-dyad figure is currently an assumption carried through the cost model, not a derivation (§7) | Low now, awkward at submission | +| C6 | Staged launch | **Declare it in the pre-registration** as an operational-only check | An undeclared mid-collection pause reads as a peek regardless of intent (#12) | **High** — cannot be declared retroactively | + +## D. Before IRB submission + +Open questions to put *to* IRB rather than settle unilaterally. See §6. + +| # | Question | Suggested position | +| --- | --- | --- | +| D1 | Withdrawal in a dyad | Most likely to come back with revisions. One participant withdrawing takes their partner's data with them, since the transcript is joint and every DV is dyad-level. Decide whether withdrawal deletes the whole dyad or retains the partner's rows — and state it in the consent text before they play | +| D2 | Post-hoc re-consent | **No.** Consent once, up front; withdrawal via the normal route. A post-hoc opt-in is a selection filter biased against exactly the sessions we most want. If IRB prefers it, offer it identically to *everyone*, and log decline rate by exit type | +| D3 | Chat as retained user-generated content | Needs an answer on abuse/self-identifying text and on researcher review before analysis | +| D4 | Display names | Assign neutral labels ("Partner A") rather than free text — cheaper than defending participant-entered PII shown to a stranger | +| D5 | Data transfer | DataPipe → OSF is third-party, likely US-hosted. A GDPR question with a UK/EU pool, and it lands on #3, which is first in the build order | +| D6 | Transcript retention | Decide retention period and whether transcripts can ever be published verbatim or only in aggregate | + +--- + +## Launch checklist + +Everything that must be set correctly **at publish time**. Each links to the reasoning rather than +restating it, so there is still one source of explanation. + +**Prolific study configuration** + +- [ ] Reward set to **$15.00/hr equivalent** for the advertised duration — B1 +- [ ] Advertised duration **includes expected lobby wait** — B5 +- [ ] Screen-out reward set from **measured** screener length, rounded up — B3 +- [ ] Screen-out slot limit set with headroom (hitting it auto-pauses the study) — B8 +- [ ] Desktop-only prescreener enabled — B7 +- [ ] Previous participants from **all** earlier runs excluded — B7 +- [ ] Study description states the partner/waiting arrangement — [`READINESS.md` §4](READINESS.md) +- [ ] Schedule-publish time chosen for a burst launch, not left open — [§4](READINESS.md) +- [ ] Places capped at the **first tranche only** (~3–4 dyads), not the full target — [#12](https://github.com/jspsych/multiplayer-test-experiments/issues/12) + +**DataPipe / OSF** + +- [ ] `limitSessions` **off** (or sized to *saves*, not participants) — B4 +- [ ] Experiment ID present in the study config block, and a test save round-trips +- [ ] Filenames carry a nonce, verified not to collide on a replayed round — A4 +- [ ] If `useValidation` is on, `requiredFields` are present on **every** row — [#4](https://github.com/jspsych/multiplayer-test-experiments/issues/4) +- [ ] Pre-registration linked to the same OSF project — [§7](READINESS.md) + +**Experiment build** + +- [ ] All four completion codes present and matching the Prolific study — [#7](https://github.com/jspsych/multiplayer-test-experiments/issues/7) +- [ ] Lobby copy's stated payment matches the configured amount (templated from one constant, not hardcoded) — B2 +- [ ] Save fires on **every** exit path: complete, partner-dropped, no-match, screened-out — [#3](https://github.com/jspsych/multiplayer-test-experiments/issues/3) +- [ ] Redirect never blocks on a save completing — [#5](https://github.com/jspsych/multiplayer-test-experiments/issues/5) +- [ ] `round_timeout` confirmed not to truncate genuine trials — [#12](https://github.com/jspsych/multiplayer-test-experiments/issues/12) + +**Governance** + +- [ ] IRB approval in hand — [§6](READINESS.md) +- [ ] Pre-registration locked, including the staged-launch declaration — C6 diff --git a/READINESS.md b/READINESS.md new file mode 100644 index 0000000..ed151d8 --- /dev/null +++ b/READINESS.md @@ -0,0 +1,543 @@ +# Prolific readiness audit + +Status of `reference-game-hawkins.html` and `reference-game-cwg.html` as candidates for a **real paid +study on Prolific**, audited 2026-07-29 against the working tree of +`chore/vendor-multiplayer-bundles` (the only branch that currently runs — `main` still points at the +broken npm `0.1.0` pins). + +Findings were verified by reading the four vendored bundles in [`vendor/`](vendor/), not just the +experiment files. Where this document contradicts a header comment in the HTML, this document is +right and the comment has been corrected. + +**Bottom line:** the experiment logic is in good shape. The gap is that there is **no pairing +mechanism, no attrition handling, and no data save target at all**. Two of those three are fixable +without waiting on upstream. + +**Decision: run `reference-game-cwg.html` as the first paid study, not Hawkins.** 6 trials (~25 min) +vs 72 trials (~55 min) changes pair-based attrition economics by roughly 3× (see +[Cost](#5-cost)), and the C&WG words-per-figure effect (41→8) is larger and more robust than +Hawkins' block-wise shortening. + +--- + +See also **[`DECISIONS.md`](DECISIONS.md)** — every open decision in one table with a suggested +answer and what it costs to reverse, plus the publish-time launch checklist. Read that first if you +want the decisions rather than the reasoning behind them. + +## Contents + +- [1. Pairing](#1-pairing) — blocked upstream, and bigger than "one script-tag swap" +- [2. Attrition](#2-attrition) — the survivor grinds, silently +- [3. Data](#3-data) — no save target exists +- [4. Prolific plumbing](#4-prolific-plumbing) — absent +- [5. Cost](#5-cost) +- [6. Ethics / IRB](#6-ethics--irb) — not started, and the longest lead time of anything here +- [7. Pre-registration](#7-pre-registration) — must be locked before the first paid participant +- [What's fine](#whats-fine) +- [Blocked on unpublished packages](#blocked-on-unpublished-packages) +- [Work plan](#work-plan) + +--- + +## 1. Pairing + +### The adapter-swap claim was false + +The header comments claimed that swapping `adapter-multiplayer-local` → +`adapter-multiplayer-firebase` required no other change to the file. At least three things change: + +1. **`SEED` breaks silently — highest-severity item in this section.** `SEED` reads `?mp_session=` + (`reference-game-hawkins.html`, `reference-game-cwg.html`), and that query param is *generated by + the LocalAdapter constructor* — confirmed in `vendor/adapter-multiplayer-local.js`, which writes + it via `history.replaceState`. Firebase will not produce it. Every dyad therefore falls through to + the literal `"cwg-fullboard"` / `"hawkins-sequential"` fallback, so **every dyad receives an + identical trial order**, confounding item order with trial number — precisely the confound the + seeding comment says the design exists to prevent. This fails with no error and produces + valid-looking, unusable data. +2. **`persistParticipant: true`** is a LocalAdapter-only constructor option. +3. `MIN_PLAYERS` gating assumes a private two-person session. Firebase provides a *shared room*, + which is a different problem — see ghost pairing below. + +### There is no waiting room + +`lobbyTrial` (`jsPsychMultiplayerSync`) is configured with **no `timeout`**. The plugin's default is +`null`, meaning `wait()` is called unbounded. A participant who arrives and is never matched **waits +on that screen forever**: no timeout, no `on_timeout`, no exit, no completion code, no payment path. + +Unmatched participants **must still be paid** — they gave up their time. Right now they cannot even +submit. + +Note that a rolling waiting room is **not** something the Firebase adapter provides. `sync` and +`role` are per-session barriers, not a matcher. Bucketing arrivals into two-person rooms, expiring +stale rooms, and handling the odd arrival is logic that has to be written on top, and it is the +largest single piece of missing work in this audit. + +### Ghost pairing + +`wait_for` counts any group entry carrying a `name`. The adapter only removes a participant's entry +on an **explicit `disconnect()`**. There is no heartbeat and no presence tracking anywhere in the +four bundles (`presence`, `heartbeat`, and `beforeunload` each return zero matches). A tab closed in +the lobby leaves its entry behind. + +With persistent Firebase room state, a live participant will be "matched" with a ghost, assigned a +role, and then time out on every single round. Silent, and frequent. + +### Spectator dead-end + +The `spectator` overflow path shows "This game is already full" for four seconds, disconnects, and +ends **with no completion code**. Anyone who is the odd person out is unpaid and unable to submit. + +--- + +## 2. Attrition + +The survivor does not hang. **They grind**, which is worse, because it consumes their time before +they give up. + +`round_timeout` sets its timer at **round start**. When it fires the round ends with +`ended_by: "timeout"` and a null assignment, feedback shows, and the timeline **advances to the next +round**, starting a fresh timer. There is no cross-round abort and no presence check: + +| | per-round dead wait | rounds left if partner drops on round 1 | total dead time | +| --- | --- | --- | --- | +| **Hawkins** | 60 s | 71 | **~71 min** | +| **C&WG** | 180 s | 5 | **~15 min** | + +And the survivor is **never told anything**. The plugin contains no partner-status copy, no +"disconnected" state, no UI affordance — searching for waiting/partner-status strings returns +nothing. They see a normal, fully interactive board that simply never responds, 72 times over. + +Realistically they abandon within minutes, so **the survivor is lost too**, and their data never +leaves the browser (§3). One dropout burns two payments *and* generates a support ticket. + +### Missing, in priority order + +- [ ] A **presence/heartbeat signal** so a client can know the partner is gone. *(needs adapter + support — see [blocked](#blocked-on-unpublished-packages))* +- [ ] A **"your partner disconnected" screen that breaks out of the game loop.** + `conditional_function` on `gameLoop` is evaluated once, so a mid-game abort needs either a + per-round `conditional_function` on `gameRound` or `jsPsych.abortCurrentTimeline()` from the + round's `on_finish` when `ended_by === "timeout"`. +- [ ] A **survivor payment path**: partial completion code, data flushed, honest debrief. Highest + value item in this section and **not blocked upstream** — two or three consecutive + `ended_by: "timeout"` rounds is a perfectly good dropout proxy, no presence API required. +- [ ] A decision on **partial-data usability**. A dyad that dies at C&WG trial 4 still has 3 clean + trials; for a 6-trial design that is worth keeping. Requires `n_trials_completed` in the data + to filter on. **This decision determines what the abort path does, so make it first.** + +--- + +## 3. Data + +### There is no save target. Cheapest catastrophic bug here. + +`initJsPsych()` is called with **no options at all** — no `on_finish`, no `on_data_update`. The only +egress in either file is a `localSave` button on `doneTrial` that the participant must voluntarily +click. No DataPipe, no OSF, no endpoint, no `fetch` anywhere in the repo. + +Consequences: + +- **Every participant who does not reach the final screen contributes zero data.** All dropouts, all + survivors who bail, all lobby timeouts, all spectators — precisely the sessions needed to + characterize attrition. +- Even completers only contribute if they click a button labelled "Download data (JSON)" on a screen + that already says "You may close this tab." Most will not. +- You are paying for data you will not receive. + +**Fix (unblocked, do first):** jsPsych DataPipe (`pipe.jspsych.org` → OSF). Wire it to +**incremental** saving, not only `on_finish` — with pair-based attrition, `on_finish`-only saving +loses exactly the sessions that matter. Save per-round, and on the abort path. + +### DataPipe's session counter is a live hazard — decide `limitSessions` deliberately + +The public docs describe only the `jsPsychPipe` plugin's `save` action, a single trial at the end of +the experiment. Incremental saving works, but the endpoint behaves in a way the docs do not mention. +Read from source (`jspsych/datapipe`, `origin/main`, `functions/src/api-data.ts`, 2026-07-30): + +**`sessions` increments on every successful save call, not once per participant.** The cap is checked +before anything else, and exceeding it returns a 400 and drops the data. So one C&WG participant +saving per round consumes **7 sessions** (6 rounds + final flush), not 1. + +| Option | Pros | Cons | +| --- | --- | --- | +| **`limitSessions` off** *(suggested)* | Cannot silently truncate a run. Recruitment is already capped by Prolific places, which is the control that actually matters. | No backstop if a bug causes runaway saves, or if the experiment ID leaks and someone else posts to it. | +| **`limitSessions` on, sized to *saves*** | Keeps a backstop against runaway saves. | Requires computing participants × (rounds + 2) with headroom, and re-computing it whenever `TRIALS` or the save granularity changes. Getting it wrong fails **silently and mid-run**. | +| **`limitSessions` on, sized to *participants*** | — | **Actively dangerous.** Looks correct, caps the run after roughly a seventh of the sample, and every save after that is lost with no participant-visible error. | + +**Suggested: leave `limitSessions` off** for the first run, since Prolific's places cap already bounds +recruitment and the failure mode of the alternative is silent mid-run data loss. If a backstop is +wanted, size it to saves and put the arithmetic on the launch checklist. + +Two further consequences for [#3](https://github.com/jspsych/multiplayer-test-experiments/issues/3), +detailed there: + +- **Filenames must be globally unique**, with a nonce rather than just a round number — an OSF 409 + is the one failure path that is *not* queued for retry, so a collision loses that round for good. +- **Everything else is retried.** `persistPending()` writes to Cloud Storage before the OSF upload, + and network/OSF errors queue and return 202. So a transient failure at abort time does not lose + data, which is why the redirect in + [#5](https://github.com/jspsych/multiplayer-test-experiments/issues/5) should never block on a save. + +### Content gaps once egress exists + +- [ ] **No `jsPsych.data.addProperties()` anywhere**, so no `PROLIFIC_PID`, `STUDY_ID`, or + `SESSION_ID` is attached to any row. Without these you cannot reconcile data against Prolific + submissions or approve payments. +- [ ] **No dyad identifier on data rows.** Subtle and important: the entire analysis is dyad-level + (shortening *within* a dyad across blocks), and nothing on a row currently says which two + participants formed the pair. `save_group: true` on `roleTrial` gives one group snapshot; + `partner_id` is in scope but never written to data. A stable `dyad_id` (session/room id) + belongs on every row. +- [ ] No `BLOCKS` / `TRIALS` / `round_timeout` config recorded, needed for provenance when piloting + at different values. +- [ ] No screen/browser/viewport capture, and **no mobile block** — a 6×2 tangram grid on a phone is + a different task. + +--- + +## 4. Prolific plumbing + +Essentially 100% absent. The only repo-wide matches for `prolific|consent|debrief|comprehension` are +prose mentions in `README.md` and the HTML header comments. + +| Required | Status | +| --- | --- | +| PID / STUDY_ID / SESSION_ID capture from URL | **missing** | +| Completion-code redirect (`app.prolific.com/submissions/complete?cc=…`) | **missing** — no participant can submit at all | +| Separate codes for complete / partner-dropped / no-match / screened-out | **missing** (all four are needed — see below) | +| Consent form | **missing** | +| Instructions | **missing** — the only task explanation is the one-line `prompt` on the game screen | +| Comprehension check | **missing** | +| Debrief | **missing** | +| Fullscreen / mobile block | **missing** | +| Timing estimate shown to participant | **missing** | + +Two items beyond the standard list that matter specifically for a *dyadic* study: + +- [ ] **Instructions must set partner expectations** — that a real person is waiting, and that + abandoning costs someone else their payment. In dyadic work this framing measurably reduces + mid-task dropout, and it is free. +- [ ] **Instructions and the comprehension check must run BEFORE the lobby.** Currently + `nameTrial` → `lobbyTrial` leaves one participant reading nothing while the other waits. + Anything slow goes before pairing. The files already apply this reasoning correctly to + `preloadTrial`, so extend it. + +Minor: + +- [ ] `nameTrial` collects a free-text display name — participant-entered PII shown to a stranger. + Assign neutral labels ("Partner A") or filter; cheaper than an IRB conversation. +- [ ] `jsPsychPreload` defaults to halting on error. One 404 in `assets/tangrams/` leaves a paid + participant dead in the water with no completion code. Set `on_error` / + `continue_after_error` deliberately. + +### Prolific's own rules for dyadic studies + +These are **platform policy, not our design choices**, and several of them constrain issues that are +already open. Recorded here so they are not rediscovered late. + +Every row cites the Prolific help-centre article it came from, all verified 2026-07-30. **Re-check +before launch** — this is policy, it changes, and Prolific's help-centre URLs are not stable, so +articles are cited by title as well as link in the [Sources](#sources) list below. + +| Rule | Consequence for us | Source | +| --- | --- | --- | +| **Prolific provides no matching infrastructure**, and explicitly does not guarantee successful pairing. Places fill first-come, first-served off ambient traffic. | The waiting room ([#10](https://github.com/jspsych/multiplayer-test-experiments/issues/10)) is entirely ours to build *and to prove*. Nothing about pairing can be assumed to work because the platform "handles it" — it does not. This is the single strongest argument for [#12](https://github.com/jspsych/multiplayer-test-experiments/issues/12) being a hard gate rather than a formality. | [D] | +| **Unmatched participants must be paid a partial payment**, minimum **$0.14 per minute** of waiting. | A *floor*, not a target — see the payment note below. Tied directly to the lobby timeout value in [#6](https://github.com/jspsych/multiplayer-test-experiments/issues/6): a 5-minute timeout implies ≥ $0.70. | [D] | +| **Unmatched participants must not be rejected.** | The approve/pay policy in [#7](https://github.com/jspsych/multiplayer-test-experiments/issues/7) is constrained, not discretionary. | [D] | +| **Expected waiting time must be included in the advertised study duration and in payment.** | The lobby timeout is not just a UX number — it sets the advertised duration. Choose it before writing the study description, not after. | [D] | +| **Identifiers must be recorded to link paired responses.** | Independent confirmation that [#4](https://github.com/jspsych/multiplayer-test-experiments/issues/4) is mandatory rather than merely advisable. | [D] | +| **In-study screening is permitted only via the built-in custom screening feature or a two-study design.** Screen-outs get a fixed reward (min $0.14), are auto-approved on a correct code, and use a screen-out slot limit that auto-pauses the study. | Screened-out participants need their **own completion code** — hence four codes, not three. | [S] | +| **The screen-out reward is duration-scaled**, not flat: $0.14 is the minimum for a screener taking **one minute or less**, so ~$0.28 at two minutes and ~$0.42 at three. | Set it from a measured pilot duration and **round up**, since it cannot be changed later. Every pre-gate item — consent, description, screening questions — counts as screening time, which is what makes brevity before the commitment gate load-bearing rather than cosmetic ([#13](https://github.com/jspsych/multiplayer-test-experiments/issues/13)). | [P] | +| **The screen-out reward may not exceed the full study reward.** | A hard cap rather than guidance. Not binding at ~$0.28 against $6.25, but worth knowing before anyone proposes a generous screen-out. | [S] | +| **The screen-out reward cannot be changed once the study is published.** | Belongs on a pre-launch checklist; it is not recoverable afterwards. | [S] | +| **Prolific cannot support time-window participation.** There is no appointment or slot-booking feature. Studies requiring completion at a particular time are explicitly *"conducted at the researcher's own risk."* The only scheduling control is **schedule publish** — choosing when the study goes live. | Kills the hope that a scheduling feature could reduce the waiting-room work in [#10](https://github.com/jspsych/multiplayer-test-experiments/issues/10). Burst launch via schedule publish is the entire simultaneity lever, and it controls when participants *can* start, not when they *do*. ([Longitudinal projects](https://researcher-help.prolific.com/en/articles/634040-how-to-run-longitudinal-studies) offer scheduled waves with guaranteed participant ordering — not slot booking, but the machinery a two-study panel approach would use if we ever wanted one.) | [T] | +| **Returned submissions are not rejections.** Participants may return a study at any time, for any reason, without penalty — Prolific advises researchers to *return rather than reject* precisely to protect a participant's approval rate. | **There is no platform filter for serial returners.** The approval-rate prescreener catches people who get rejected, not people who abandon. Mid-task abandonment — the failure mode that costs us two payments and a dyad — is therefore invisible to Prolific's screening and has to be addressed in our own design (§4 pre-lobby framing, [#13](https://github.com/jspsych/multiplayer-test-experiments/issues/13)) and survived by the abort path ([#5](https://github.com/jspsych/multiplayer-test-experiments/issues/5)). | [R] | + +**Suggested (tentative) consequence for payment design: pay the no-match exit at the full task rate, +not at the $0.14/min floor.** The floor is badly misaligned with participants' actual incentives. A +participant who has waited and been paid $0.70 for it can instead return the study at no penalty and +take a two-minute survey for more — so a floor-rate no-match payment actively teaches people to +abandon the lobby the moment matching feels slow, which is exactly when we most need them to stay. + +At the §5 rate of $15.00/hr, paying the wait at the task rate makes a 5-minute no-match exit $1.25 +(≈$1.66 all-in) instead of $0.70. Because unmatched participants are a small fraction of the total, +the absolute cost is on the order of $10–15 across a full run — negligible against the pairing yield +it buys. + +This does not create an incentive to prefer being unmatched, because participants cannot influence +whether they match. There is one adjacent case worth being precise about: once the commitment gate +([#13](https://github.com/jspsych/multiplayer-test-experiments/issues/13)) exists, someone who does +not intend to complete the study *can* choose between taking the screen-out exit now and entering the +lobby hoping to go unmatched. That gamble still loses — it costs up to five minutes of waiting to +maybe win $1.25, against a certain payment available immediately for zero minutes, and a match leaves +them choosing between 25 unwanted minutes and a return for nothing. The sure thing dominates, which +is why the screen-out reward does **not** need to be inflated to compete with the no-match payment. + +#### Sources + +All Prolific researcher help-centre articles, verified 2026-07-30: + +- **[D]** [Can I run dyadic studies on Prolific?](https://researcher-help.prolific.com/en/articles/445177-dyadic-studies) +- **[S]** [How to use custom screening to recruit specific participants](https://researcher-help.prolific.com/en/articles/445155-how-to-use-custom-screening-to-recruit-specific-participants) +- **[P]** [How much should I pay participants?](https://researcher-help.prolific.com/en/articles/445266-how-much-should-i-pay-participants) +- **[R]** [Approving, rejecting, and returning submissions](https://researcher-help.prolific.com/en/articles/445208-approving-rejecting-and-returning-submissions) +- **[T]** [What if my study is time sensitive?](https://researcher-help.prolific.com/en/articles/445150-what-if-my-study-is-time-sensitive) +- **[M]** [Prolific's payment model](https://researcher-help.prolific.com/en/articles/445230-prolific-s-payment-model) — the $8.00/hr minimum and $12.00/hr recommended rates used in §5 +- [Can I make partial payments to participants?](https://researcher-help.prolific.com/en/articles/445234-can-i-make-partial-payments-to-participants) + +--- + +## 5. Cost + +All amounts are in **USD**, since that is the currency participants are paid in. Prolific's own +thresholds are $8.00/hr minimum and $12.00/hr recommended ([source](https://researcher-help.prolific.com/en/articles/445230-prolific-s-payment-model), verified 2026-07-30). + +**Suggested (tentative) rate: $15.00/hr**, above Prolific's recommendation. The task is 25 minutes of +sustained real-time coordination with another person and no ability to pause, and attention *is* the +measurement here. Rate also partly pays for itself: dropout is multiplicative on dyads, so if better +pay moves per-participant dropout from 15% to 10%, dyad survival goes 0.72 → 0.81 and the number of +dyads you must start falls from ~28 to ~25 — recovering roughly a third of the extra cost. + +A 33% Prolific service fee applies on top of participant pay throughout. + +**C&WG** (~25 min: instructions + comprehension + lobby + 6 trials) → $6.25/participant, +**$8.31 all-in**. Attrition is multiplicative on dyads; at ~15% per-participant mid-task dropout +(reasonable for a 25-min synchronous task with good framing), dyad survival ≈ 0.72: + +| | for 20 usable dyads | +| --- | --- | +| dyads to start | ~28 | +| participants recruited | ~56 | +| of which usable | 40 | +| **paid but unusable** | **~16** (dropouts + their stranded survivors) | +| **cost** | **~$465** | + +Add the no-match exits (~10–15% of participants at ~$1.66 all-in each, so roughly $10–15 — see §4), +plus headroom for one wasted session while learning real arrival-rate and attrition numbers. +**Budget ~$580 for the first real run.** + +**Hawkins** (~55 min) → $13.75/participant, **$18.29 all-in**, and 72 synchronous trials realistically +push per-participant dropout to ~30%, so dyad survival ≈ 0.49 → ~41 dyads → ~82 participants → +**~$1,500** for the same 20 dyads. Three-plus times the cost, with the §2 survivor-grind failure mode +at its worst. + +### Two structural cost levers, both large + +- **Simultaneous arrival is the real constraint.** Pair-based studies need participants + *concurrently*, which Prolific does not guarantee. You control it by launching in a tight burst + rather than leaving the study open — which is why the waiting room and its timeout policy are + load-bearing on cost, not just on UX. +- **Partial-data salvage.** With C&WG's 6 trials, a dyad dying at trial 4 yields 3 usable trials. If + the analysis can use partial dyads, effective attrition cost drops substantially. + +--- + +## What's fine + +Worth stating explicitly, because the list above is long: + +- **The replication logic itself.** Both files were verified parameter-by-parameter against the + papers and against `hawkrobe/tangrams`, and the header comments documenting those decisions are + accurate and unusually thorough. +- **The per-round data spec** covers every stated DV: `message_count`, `messages_sent`, + `chat_transcript`, `accuracy`, `n_correct`, `correct`, `rt`, `my_order` / `partner_order`, + `ended_by`, plus `interaction_history` (enabled in both files). Analysis-side, once data reaches + you, you are in good shape. +- **`ended_by: "timeout"` + null assignment** behaves exactly as the header comments describe; + verified in the plugin's finish path. The documented analysis filter works. +- **`chat_transcript` is built from `getAll()`** — the *full* shared transcript, not just the local + client's messages. A survivor's saved rows therefore contain the complete dialogue up to the drop, + including the departed partner's messages. **The core DV survives a dropout**, provided the data + can get out (§3). +- **Preload runs before pairing**, correctly, so nobody waits on someone else's image fetches. +- **`vendor/` pinning is correct and internally consistent.** Bundles built from `69c0d7b`, loaded + from commit `5046bf0`, and both HTML files pin `5046bf0`. The stale-pin hazard `vendor/README.md` + warns about is real but not currently triggered. + +> **Do not "fix" the `vendor/` pins back to npm.** The published `0.1.0` builds predate the +> `jsPsych.multiplayer` namespace migration and do not work with the pinned core. `main` currently +> has exactly this bug, having reverted the vendoring. See [`vendor/README.md`](vendor/README.md). + +--- + +## Blocked on unpublished packages + +Tracked separately because these cannot be unblocked from this repo. + +1. **`adapter-multiplayer-firebase` is unpublished** → no cross-device pairing at all. Everything in + §1 and §2 is untestable end-to-end until this lands. **This is the hard gate on a paid run.** +2. **No presence/disconnect API in any bundle**, confirmed by inspecting all four. The proper fix for + §2 — *knowing* the partner is gone rather than inferring it from timeouts — needs either the + Firebase adapter exposing presence (Firebase RTDB has `onDisconnect`, so it plausibly does or + could) or an upstream plugin change. **Verify this specifically when the adapter is available**; + it is the difference between a good survivor experience and a tolerable one. The timeout-counter + workaround in §2 does not depend on it. +3. **`plugin-multiplayer-reference-game` is unpublished**, and the other three have only + pre-namespace-migration `0.1.0` releases. Gated on + [jspsych-multiplayer PR #35](https://github.com/jspsych/jspsych-multiplayer/pull/35). + +--- + +## 6. Ethics / IRB + +Not started. It belongs in this document because IRB approval is a hard blocker on a paid run, and +it has the **longest lead time of anything listed here** — every other item is days of work, this one +is weeks of someone else's queue. Starting it late is what actually delays a launch. + +The generic protocol items (consent, withdrawal, data handling) are assumed. What follows is the +short list of things that are unusual about *this* study and that a reviewer will stop on. All of it +is a suggested starting point for the submission, not a settled position. + +- **Free-text chat between strangers is unmoderated user-generated content.** Two anonymous + participants type arbitrary text to each other in real time, and we retain it verbatim because + `chat_transcript` *is* the primary DV. Most single-participant studies never have this + conversation. Expect to be asked what happens if someone types something abusive or + self-identifying, and to have an answer about reporting and about researcher review of + transcripts before analysis. +- **Participant-entered display names are shown to a stranger.** `nameTrial` currently collects + free text and renders it to the partner. Assigning neutral labels ("Partner A") removes the + problem outright and is cheaper than defending it — see + [#13](https://github.com/jspsych/multiplayer-test-experiments/issues/13). +- **Right to withdraw is genuinely complicated in a dyad**, and this is the question most likely to + come back with revisions. One participant withdrawing takes their partner's data with them: the + transcript is jointly produced and every DV is dyad-level, so there is no clean way to remove one + person's contribution. Decide in advance whether withdrawal deletes the whole dyad (defensible, + expensive) or retains the partner's rows under the original consent (cheaper, needs justifying in + the consent text itself). Whichever is chosen has to be stated to participants *before* they play. +- **Data transfer.** DataPipe → OSF means participant text leaves for third-party, likely US-hosted + infrastructure. With a UK/EU participant pool that is a GDPR question, and it lands on + [#3](https://github.com/jspsych/multiplayer-test-experiments/issues/3), which is first in the + build order — so confirm the hosting position before building on it rather than after. +- **Transcript retention and release.** Free text can contain anything; decide the retention period + and whether transcripts can ever be published as-is or only in aggregate. +- **Payment policy for participants who are screened out, unmatched, or stranded by a partner's + dropout.** Prolific's own rules already constrain this (§4, §5): unmatched participants must be + paid and must not be rejected. Stating the policy in the protocol up front avoids a mismatch + between what IRB approved and what the completion codes in + [#7](https://github.com/jspsych/multiplayer-test-experiments/issues/7) actually do. +- **Question to put to IRB: should data use be re-consented after the fact?** Suggested position is + **no** — consent is taken once, up front, and withdrawal runs through the normal route. The reason + is that a post-hoc opt-in is a selection filter: stranded survivors are likelier to decline than + completers, and being stranded correlates with the DV (longer, more effortful sessions), so + re-consent would preferentially delete the high end of the measure. Post-hoc re-consent *is* + standard for **deception** studies, where the debrief reveals that the original consent was not + fully informed; this study is not one, so that rationale does not carry over. If IRB prefers + re-consent anyway, it must be offered **identically to every participant**, not only to those + whose partner left — and note that identical wording still does not equalise decline *rates* + between groups, so log the decline rate by exit type to keep the bias measurable. + +**Suggested sequencing:** draft the protocol in parallel with phase 1 rather than after it. The +consent and debrief text is needed by #13 anyway, and writing it early surfaces the withdrawal and +retention decisions while they are still cheap to act on. + +--- + +## 7. Pre-registration + +Not started, and it has a **hard deadline that nothing else here has**: a pre-registration must be +locked *before the first paid participant*, not before analysis. That makes it the real deadline on +the [#8](https://github.com/jspsych/multiplayer-test-experiments/issues/8) analysis decision, which +is otherwise the most deferrable item in the work plan. + +OSF is the natural venue, and the data already lands there via DataPipe (§3) — register the +pre-registration against the same OSF project so the two are linked rather than discovered separately +later. + +### What it requires that we do not currently have + +- [ ] **Sample size with a justification.** §5 works throughout to "20 usable dyads", but that number + is an *assumption carried through a cost model*, not a derivation. Nothing in this repo powers + it. A pre-registration needs a power analysis, or an explicit and defensible statement that the + target is resource-constrained rather than power-derived — the latter is acceptable and honest, + but it has to be said rather than implied. +- [ ] **The analysis plan**, i.e. the #8 decision: mixed-effects with trial as a continuous predictor + versus a trial-1-vs-final contrast, and whether partial dyads enter the primary analysis or + only a robustness check. +- [ ] **Exclusion criteria**, stated in advance: minimum completed trials for a partial dyad, prior + tangram exposure, comprehension-check failure, and dyads ended by the #5 abort path. +- [ ] **Hypotheses and DVs** — words/messages per figure across trials, accuracy — which the file + header comments already state clearly and can largely be lifted from. +- [ ] **A stopping rule.** This is the one most likely to be missed. Powering for completers-only + (§5) exists precisely so that topping up after seeing data is never tempting, because that is + optional stopping. + +### The staged launch must be declared + +[#12](https://github.com/jspsych/multiplayer-test-experiments/issues/12) proposes launching ~3–4 +dyads, pausing to inspect, then releasing the rest. That is sound practice, but an undeclared pause +mid-collection looks exactly like a peek — so the pre-registration should state that the pause +happens, that its checklist is **operational only**, and that the decision it gates is "does the +instrumentation work" rather than anything about the DV. Declared in advance it is a quality control +step; discovered afterwards it is a credibility problem. + +--- + +## Work plan + +Everything in phase 1 is unblocked, and it is most of the total effort. + +### Agreed execution order + +The phases below group by *upstream dependency*, which is not the same as the order to work in. +The order we settled on, and why: + +1. **#8** — decide the partial-data policy. No code; determines whether #5 salvages or just exits, + and whether #3 must save per-round or only on abort. Do it before writing either. +2. **#3** — data egress. Nothing else is observable without it, and #5 needs a flush target. +3. **#4** — identifiers. Same code region as #3; near-worthless before it, near-free after. +4. **#7** — completion codes + redirect. Unblocks the payment paths in #5 and #6. +5. **#5** — dropout abort. Highest attrition value of the unblocked work, and by now it has both a + flush target and an exit to redirect to. +6. **#6** — lobby timeout. Reuses #5's exit screens, so it is cheap once those exist. +7. **#13** — the participant-facing wrapper. Largest unblocked chunk, but depends on nothing: + start it in parallel any time after step 1. + +Then the phase-2 gate: **#11** before **#10**, because whether presence is exposed changes how ghost +expiry gets built (`onDisconnect` tombstones if yes, a staleness sweep if no). **#9** ships in the +same commit as the adapter swap. **#12** last. + +**Do phase 1 against `reference-game-cwg.html` only.** The two files are near-duplicates and every +phase-1 item applies to both, but C&WG is the first paid run (§5) and back-porting to Hawkins before +the pilot means writing every fix twice against assumptions the pilot may invalidate. Factoring the +shared plumbing into a common module is the right end state, but two concrete implementations are a +better basis for that extraction than one speculative one. Re-cost Hawkins only after #12 replaces +the assumed 15% / 30% dropout figures in §5 with observed ones. + +### Phase 1 — now, no upstream dependency + +- [ ] [#3](https://github.com/jspsych/multiplayer-test-experiments/issues/3) — **Data egress** + (DataPipe/OSF, incremental saves). Highest value per line of code in this document; without it + every other fix produces nothing analyzable. +- [ ] [#4](https://github.com/jspsych/multiplayer-test-experiments/issues/4) — **Dyad + Prolific + identifiers** on every row via `addProperties` (`dyad_id`, `PROLIFIC_PID`, `STUDY_ID`, + `SESSION_ID`). +- [ ] [#5](https://github.com/jspsych/multiplayer-test-experiments/issues/5) — **Timeout-based + dropout detection + abort path** (N consecutive `ended_by: "timeout"` → flush data → "partner + disconnected" screen → partial completion code). Kills the 71-minute grind with no presence API. +- [ ] [#6](https://github.com/jspsych/multiplayer-test-experiments/issues/6) — **Lobby timeout + + no-match exit + payment path**, and fix the spectator dead-end. +- [ ] [#7](https://github.com/jspsych/multiplayer-test-experiments/issues/7) — **Completion codes + + submission redirect** (three codes: complete / partner-dropped / no-match). Small, and a + blocking dependency of both #5 and #6 — neither payment path can be built without it. +- [ ] [#13](https://github.com/jspsych/multiplayer-test-experiments/issues/13) — **Prolific + wrapper**: consent, instructions with partner-expectation framing, comprehension check, + debrief, mobile block. All before the lobby. Split out of #7; depends on nothing. +- [ ] [#8](https://github.com/jspsych/multiplayer-test-experiments/issues/8) — **Decide the + partial-data policy.** It changes what #5 does, so settle it early. + +### Phase 2 — after `adapter-multiplayer-firebase` publishes + +- [ ] [#9](https://github.com/jspsych/multiplayer-test-experiments/issues/9) — **Fix `SEED` + derivation for Firebase** (§1). Do this *in the same change* as the adapter swap, or you will + collect a wave of confounded data that looks fine. +- [ ] [#10](https://github.com/jspsych/multiplayer-test-experiments/issues/10) — Build the + **rolling waiting room** (room bucketing, stale-room expiry, odd-arrival handling). Largest + remaining piece. +- [ ] [#11](https://github.com/jspsych/multiplayer-test-experiments/issues/11) — **Verify whether + presence is exposed**; if so, upgrade #5 from timeout-inference to real disconnect detection. + +### Phase 3 + +- [ ] [#12](https://github.com/jspsych/multiplayer-test-experiments/issues/12) — **Two-device pilot + with real strangers**, deliberately including a scripted mid-round dropout, before spending + money. diff --git a/README.md b/README.md index 2407807..af7e13e 100644 --- a/README.md +++ b/README.md @@ -45,13 +45,20 @@ These currently run on `adapter-multiplayer-local` (no backend needed) for two-t into a second tab so a second player joins. 4. The first tab becomes the director, the second the matcher. -For a paid Prolific run, swap `adapter-multiplayer-local` for -[`adapter-multiplayer-firebase`](https://github.com/jspsych/jspsych-multiplayer/tree/main/packages/adapter-multiplayer-firebase) -(one script-tag swap, see the header comment in each file) plus a real waiting room for pairing. +## Running it as a paid study -**Note on package versions:** the `@jspsych-multiplayer/*` package script tags below are pinned to -`0.1.0` on jsDelivr, but those packages are not yet published to npm — publishing is gated on +**Not yet possible — see [`READINESS.md`](READINESS.md)** for a full audit. In short: there is no +pairing mechanism for strangers, no partner-dropout handling, and no data save target, and +cross-device pairing is gated on the unpublished +[`adapter-multiplayer-firebase`](https://github.com/jspsych/jspsych-multiplayer/tree/main/packages/adapter-multiplayer-firebase). + +Swapping the adapter is **not** a one-script-tag change: `SEED` is derived from the `?mp_session=` +param that the *local* adapter generates, so on Firebase every dyad would silently receive the same +trial order. `READINESS.md` lists what actually has to change, and which items are blocked upstream. + +**Note on package versions:** the four `@jspsych-multiplayer/*` bundles load from +[`vendor/`](vendor/README.md) in this repo, not npm — `plugin-multiplayer-reference-game` is +unpublished and the other three have only pre-namespace-migration `0.1.0` releases, which do not work +with the pinned core. Publishing is gated on [jspsych-multiplayer PR #35](https://github.com/jspsych/jspsych-multiplayer/pull/35) ("Version -Packages") merging. Until then, either build the packages from a local checkout of -jspsych-multiplayer and swap in relative `dist/` paths, or wait for the publish and confirm the -pinned version still matches. +Packages"). Do not repoint these to npm until that merges. diff --git a/reference-game-cwg.html b/reference-game-cwg.html index ef884c5..25f22da 100644 --- a/reference-game-cwg.html +++ b/reference-game-cwg.html @@ -8,9 +8,12 @@ - + +