diff --git a/reference-game-cwg.html b/reference-game-cwg.html index 7a54a72..59730fb 100644 --- a/reference-game-cwg.html +++ b/reference-game-cwg.html @@ -103,6 +103,34 @@ // every participant who does not click the download button contributes nothing. DATAPIPE_EXPERIMENT_ID: "", + // --- Dropout detection (#5) --------------------------------------------------------------- + // Consecutive SILENT rounds before concluding the partner is gone. A round counts as silent + // only if it timed out AND the partner sent no chat message during it — see the detector for + // why both conditions are needed. Two is the suggested value: at a 180s round that is ~6 + // minutes of dead time before the survivor is released, against 15 minutes today, while still + // requiring corroboration rather than firing on a single slow round. + // + // The trade is asymmetric, and NOT in the direction it first looks. A false negative leaves a + // survivor grinding; a false positive ends a live game for BOTH participants and pays two + // partial codes — the same cost as the dropout this is guarding against, inflicted on a dyad + // that was working. Requiring partner silence is what makes 2 tolerable: a slow dyad that is + // still talking is never aborted. + // + // The known false-positive channel, stated plainly because the guard above does not close it: + // silence is not absence. A matcher can be actively working — clicking slots, placing 12 + // tangrams — and typing nothing, and `save_interaction_history` is matcher-local until submit, + // so from the director's side that is byte-identical to a partner who has gone. Two slow first + // rounds on a full board is not far-fetched, and it is exactly where the effect is largest. + // + // Pilot check (#12): if `partner_dropped` rows cluster at `n_trials_completed` 0-1, that is + // this false positive, not a real dropout rate. The fix then is a POSITIVE-EVIDENCE signal — + // surfacing the matcher's slot activity into the shared session, which needs a channel the + // bundles do not currently expose (#11) — and NOT raising this number. A higher threshold buys + // safety with dead minutes and does not resolve the ambiguity: an active silent matcher looks + // absent at 3 rounds too, just later. That is the same trade the detector already refuses when + // it declines to count bare timeouts. + DROPOUT_SILENT_ROUNDS: 2, + // A4 — save per-round chunks rather than once at the end. A single end-of-run save loses // everything for any dyad that breaks, which is exactly the population #12 needs to // characterise. Assumes `limitSessions` is OFF (B4): DataPipe's counter increments per SAVE @@ -502,6 +530,70 @@ let myRole; let partnerId; + // =============================================================================================== + // DROPOUT DETECTION (#5) + // + // There is no presence API in any of the four bundles, so the partner's absence has to be + // inferred. The inference has to be good, because acting on it ends a paid session. + // + // `round_timeout` alone is NOT the signal. It is a wall-clock bound (see CONFIG), so it fires + // identically for a partner who left and a pair who are simply slow — and trial 1 of a 12-figure + // board is the longest in the study, so a genuine timeout there is entirely normal. Counting + // timeouts alone would abort real dyads at exactly the point the effect is largest. + // + // So a round is "silent" only if it timed out AND the partner sent nothing during it. The plugin + // records `message_count` (every message that round) and `messages_sent` (mine), and the chat + // channel is per-round, so the difference is the partner's activity in that round alone. A + // partner who is talking is present, however slow they are. + // + // TWO CONFIG DEPENDENCIES, both currently satisfied by defaults and both silent if broken. There + // are tests for each in tests/dropout.test.mjs, because neither fails loudly at runtime: + // + // 1. `chat_persists` must stay OFF. The plugin keys the chat channel `_chat_r` + // when it is off and `_chat` when it is on, so turning it on makes `message_count` + // CUMULATIVE: `partnerMessages` never returns to zero, the detector never fires again, and + // the survivor grinds — the exact failure this exists to prevent, arrived at silently. + // 2. `ended_by: "timeout"` is only equivalent to "the round clock ran out" because + // `selection_timeout` is unset. The plugin emits the same string for a selection timeout, + // which is a different event with a possibly-present partner. + // + // Upgrade path: when a presence API exists (#11), this becomes a fallback rather than the + // primary signal. It does not need to wait for one. See CONFIG for the false-positive channel + // this cannot close — an active but silent matcher — and what to do if the pilot shows it. + // =============================================================================================== + let consecutiveSilentRounds = 0; + let partnerDropped = false; + let dropoutRound = null; + + // Trials the dyad actually completed, as opposed to rounds the timeline advanced through. Only + // `ended_by: "submit"` counts — a timed-out round has a null assignment and is not a trial. + // This is what makes a partial dyad usable rather than merely present (#8). + const completedTrials = () => + jsPsych.data + .get() + .filter({ trial_type: "multiplayer-reference-game", ended_by: "submit" }) + .count(); + + // Session-level facts, stamped onto every row at the point the game ends by any route. Recorded + // here rather than left to analysis because "how did this session end" cannot be reconstructed + // reliably from partial data — which is precisely the data most likely to be all we have. + // + // First writer wins. `addProperties` applies retroactively to every row already collected, so a + // second call would not merely add a row — it would rewrite the FIRST outcome on every existing + // row and destroy the reason the session actually ended. Exactly one outcome is recorded per + // session today; this makes that structural rather than a property of the current routing. + let endedReason = null; + function recordOutcome(reason) { + if (endedReason) return; + endedReason = reason; + jsPsych.data.addProperties({ + ended_reason: reason, + n_trials_completed: completedTrials(), + n_trials_scheduled: TRIALS, + dropout_detected_at_round: dropoutRound, + }); + } + const nameTrial = { type: jsPsychSurveyText, questions: [{ prompt: "Choose a display name:", required: true, name: "name" }], @@ -546,6 +638,7 @@ on_finish: () => { // Flush even here. A spectator has no game data, but their arrival and the fact they were // turned away is the raw material for the odd-arrival rate #10's waiting room must handle. + recordOutcome("spectator"); if (CONFIG.FLUSH_ON_ABORT) Pipeline.flush("spectator"); jsPsych.multiplayer.disconnect(); }, @@ -583,6 +676,23 @@ // Deliberately not awaited: the next round must not wait on OSF. on_finish: (data) => { if (CONFIG.SAVE_PER_ROUND) Pipeline.save(`round-${data.round ?? "?"}`); + + // Partner activity in THIS round. `message_count` is every message on the round's channel + // and `messages_sent` is mine, so the difference is theirs. A round that timed out while the + // partner was still talking is a slow round, not a dropout. + const partnerMessages = (data.message_count ?? 0) - (data.messages_sent ?? 0); + const silent = data.ended_by === "timeout" && partnerMessages <= 0; + consecutiveSilentRounds = silent ? consecutiveSilentRounds + 1 : 0; + + if (!partnerDropped && consecutiveSilentRounds >= CONFIG.DROPOUT_SILENT_ROUNDS) { + partnerDropped = true; + dropoutRound = data.round ?? null; + recordOutcome("partner_dropped"); + // `conditional_function` on gameLoop is evaluated ONCE, before the first round, so it + // cannot end the loop mid-game. Aborting the enclosing timeline from here is what actually + // stops the remaining rounds; execution resumes at the screens after gameLoop. + jsPsych.abortCurrentTimeline(); + } }, prompt: (role) => role === "director" @@ -596,6 +706,46 @@ conditional_function: () => myRole === "director" || myRole === "matcher", }; + // The survivor's exit (#5). Honest about what happened, explicit that they are still paid, and + // it never blames them. This screen is the entire difference between a survivor who submits and + // one who abandons and opens a support ticket. + const partnerDroppedScreen = { + timeline: [ + { + type: jsPsychHtmlKeyboardResponse, + stimulus: () => { + const played = completedTrials(); + return `

Your partner seems to have disconnected

+

We waited, but your partner stopped responding, so we have ended the study here. + This is not your fault, and your payment is not affected.

+

You completed ${played} of ${TRIALS} round${played === 1 ? "" : "s"} together, and + ${played > 0 ? "that work is saved and still useful to us" : "your time is still paid"}.

+

Saving your responses…

+ ${submissionBlockHTML("partner_dropped")}`; + }, + choices: "NO_KEYS", + on_load: () => { + // Wire the button FIRST. The participant must be able to submit immediately, whatever + // the save is doing — the whole point of this screen is that they get paid. + wireSubmissionButton("partner_dropped"); + jsPsych.multiplayer.disconnect(); + Pipeline.flush("abort-partner-dropped").then((r) => { + const el = document.getElementById("save-status"); + if (!el) return; + el.textContent = r.skipped + ? "" + : r.ok + ? "Your responses have been saved." + : r.timedOut + ? "Still uploading — you can submit now, this will finish in the background." + : "Your responses could not be uploaded. Please submit anyway and message the researcher."; + }); + }, + }, + ], + conditional_function: () => partnerDropped, + }; + const doneTrial = { type: jsPsychHtmlKeyboardResponse, stimulus: () => { @@ -618,6 +768,7 @@ choices: "NO_KEYS", on_load: () => { wireSubmissionButton("complete"); + recordOutcome("complete"); // Final flush: the last round's rows, plus anything an earlier chunk failed to send. Fired // on_load rather than on_finish because this trial takes no keys and never finishes — the @@ -650,6 +801,17 @@ }, }; + // `doneTrial` used to run unconditionally, which meant two people who never finished the game + // still reached "Game complete!" and were handed the `complete` code: the survivor of a dropout, + // and — already, before this change — any spectator, who saw a 4-second "game is full" screen and + // then a completion screen reporting 0 / 0. Both are paid under the wrong code, which defeats the + // point of having four, and the spectator case also contradicts what the screen before it said. + const completeScreen = { + timeline: [doneTrial], + conditional_function: () => + !partnerDropped && (myRole === "director" || myRole === "matcher"), + }; + jsPsych.multiplayer.connect(localAdapter).then(() => { // Preload FIRST, before pairing: a participant who is still fetching images while their // partner waits in the lobby wastes the partner's time. @@ -660,7 +822,10 @@ roleTrial, spectatorScreen, gameLoop, - doneTrial, + // Exactly one of these two runs, and each carries its own completion code. gameLoop's own + // conditional cannot end it mid-game, so the abort comes from inside the round (#5). + partnerDroppedScreen, + completeScreen, ]); }); diff --git a/tests/dropout.test.mjs b/tests/dropout.test.mjs new file mode 100644 index 0000000..a76273a --- /dev/null +++ b/tests/dropout.test.mjs @@ -0,0 +1,170 @@ +// Tests for dropout detection (#5) in reference-game-cwg.html. +// +// node tests/dropout.test.mjs +// +// Extracts the real `on_finish` body from the gameRound definition and drives it with synthetic +// round data, so the detector is tested as written rather than as re-implemented here. +// +// The thing under test is a judgement call with an asymmetric cost: a false positive ends a live +// dyad and pays two partial codes, a false negative leaves a survivor grinding. The cases below are +// mostly about the false-positive side, because that is the one that silently destroys good data. + +import fs from "fs"; + +const html = fs.readFileSync(new URL("../reference-game-cwg.html", import.meta.url), "utf8"); +const scriptMatch = html.match(/