Skip to content

Align the cued reference game with the original Hawkins et al. experiment - #19

Open
Mandyx22 wants to merge 1 commit into
mainfrom
hawkins-cued-fidelity
Open

Align the cued reference game with the original Hawkins et al. experiment#19
Mandyx22 wants to merge 1 commit into
mainfrom
hawkins-cued-fidelity

Conversation

@Mandyx22

Copy link
Copy Markdown
Collaborator

This is the build we are using for the paper. reference-game-hawkins.html is the study artifact of record for the cued (sequential) reference game — the parameters here are the ones the reported data will be collected under, so changes to it are changes to the protocol.

What "cued" means

The OSF project behind Hawkins, Frank & Goodman (2020) (osf.io/vzvmf) attaches hawkrobe/tangrams as its github component. The paper ran two variants:

  • free-matching — the Clark & Wilkes-Gibbs style drag-the-board replication (experiments/tangrams_unconstrained)
  • cued — "a more straightforwardly sequential cued variant of the task design where directors were privately cued to refer to targets one-by-one and feedback was given on each trial" (experiments/tangrams_sequential)

The paper restricts its analyses to the cued version "as a cleaner confirmatory sample," and osf.io/vzvmf is that version's preregistration. This PR audits our cued build line by line against that source and the manuscript.

Changed to match the original

  • Stimulus geometry. game.core.js lays out a 6×2 grid of square 300×300px cells, and stimuli/objectSet.js draws every tangram at exactly 0.7× its natural PNG size. The shapes are deliberately not normalised — one uniform scale preserves their relative sizes. Cells are now square and each image gets width = originalDrawWidth / 300 as a percentage of the cell content box, exact at any cell_size.
  • scramble_mode: "disjoint". The original re-rolls both layouts until no tangram shares a cell between the two players (notMatchingLocs / arraysDifferent). Plain "independent" only guarantees the layouts differ, leaving ~1/e of tangrams coinciding by chance — enough that positional reference sometimes worked by luck.
  • Role-keyed feedback. Per the paper, "the director saw which tangram their partner clicked, and the matcher saw the intended tangram." Each role now sees exactly one thing. CSS also repaints the plugin's red "wrong" highlight green, since the original highlights green either way and carries the outcome only through the score counter.
  • Persistent round counter and running score above the board, matching the original's #roundnumber / #score.
  • Instruction wording from client_onjoingame. One edit: the original tells both roles they may "talk to the director," a copy-paste slip, so each role now names the other.
  • Typing indicator — "Participants saw a binary indicator that their partner was currently typing." Built on the multiplayer API from the experiment file, not a plugin fork.

Already correct, left alone

Two players, fixed director/matcher, 6×2 grid, 72 trials as 6 shuffled blocks of 12, per-trial re-scrambling, cued target, unrestricted two-way chat, per-round chat log, the messageSent click gate, the 3000 ms feedback window, and unnumbered cells (the paper removed the numbers for the cued version; the plugin only draws slot badges when there is more than one target).

Known differences, documented in the file header

  1. Gate semantics. The original sets messageSent in its chatMessage handler, which fires for any message including the matcher's own, so a matcher could unlock their own click by typing anything. Ours counts only the partner's message — the stricter reading, and what that code's own comment says was intended.
  2. No comprehension quiz. The original gates entry on "a short quiz about task instructions." Worth adding before data collection.
  3. round_timeout (60s) has no counterpart in the original, which ran untimed. It exists so a disconnected partner cannot hang a trial. Rounds it ends record ended_by: "timeout" with a null assignment, so they filter cleanly.

Dependency — read before merging

This needs two options that do not exist in a published plugin yet: scramble_mode: "disjoint" and role-keyed feedback_content. Both are on the reference-game-cued-fidelity branch of jspsych-multiplayer (12 new tests, suite green). That has to merge and publish first.

Until it does, reference-game-hawkins.html cannot run at all — its CDN tag for plugin-multiplayer-reference-game 404s, because the package has never been published to npm. So this PR also adds reference-game-hawkins-local.html, an identical build that loads from a gitignored vendor/ directory; its header carries the rebuild commands. Once the plugin publishes, pin the version in the main file and delete the local twin.

Testing

Structural verification only: both files parse, their timelines are byte-identical, all referenced resources resolve, and a jsdom render of a real trial confirms the round counter and score, both roles' wording, 12 images with src values the per-image CSS matches, the square grid tracks, and the click gate blocking before the director's message and submitting after.

Not yet verified in a browser. The stimulus geometry, the green repaint, the per-role feedback and the typing hint are all visual and nobody has watched them render. A two-tab pass through a few rounds should come before this merges.

🤖 Generated with Claude Code

Audited reference-game-hawkins.html against the experiment code and manuscript
behind the paper: OSF osf.io/vzvmf -> github component = hawkrobe/tangrams. The
paper's two variants are "free-matching" (the Clark & Wilkes-Gibbs style board
replication, experiments/tangrams_unconstrained) and "cued" -- "a more
straightforwardly sequential cued variant ... where directors were privately cued
to refer to targets one-by-one and feedback was given on each trial"
(experiments/tangrams_sequential). This file is the cued one.

Changed to match:

- Stimulus geometry. game.core.js lays out SQUARE 300x300px cells and
  stimuli/objectSet.js draws every tangram at exactly 0.7x its natural PNG size,
  so the shapes are NOT normalised -- one uniform scale preserves their relative
  sizes. Cells are square and each image gets width = originalDrawWidth/300 as a
  percent of the cell content box, exact at any cell_size.

- scramble_mode: "disjoint" (new plugin option). The original re-rolls both
  layouts until no tangram shares a cell between players; plain "independent" left
  ~1/e of them coinciding, so positional reference sometimes worked by luck.

- Role-keyed feedback_content (new plugin option): the director sees only the
  object the matcher clicked, the matcher only the true target, per the paper --
  "the director saw which tangram their partner clicked, and the matcher saw the
  intended tangram". CSS repaints the plugin's red "wrong" highlight green, since
  the original highlights green either way and carries the outcome only through
  the score counter.

- Persistent round counter and running score above the board, matching the
  original's #roundnumber/#score, with show_running_score off.

- Instruction wording from client_onjoingame. Only edit: the original tells both
  roles they may "talk to the director", a copy-paste slip, so each role now names
  the other.

- Typing indicator, built on the multiplayer API rather than forking the plugin.
  The original has one ("Participants saw a binary indicator that their partner
  was currently typing"; sharedUtils/clientBase.js playerTyping).

Already correct and left alone: 2 players, fixed roles, 6x2 grid, 72 trials as 6
shuffled blocks of 12, per-trial re-scrambling, cued target, unrestricted two-way
chat, per-round chat log, the messageSent click gate, the 3000ms feedback window,
and unnumbered cells (the paper removed the numbers for the cued version; the
plugin only draws slot badges when there is more than one target).

Two documented differences remain, in the file header: gate semantics (the
original's gate opens on any message, including the matcher's own) and the absence
of the original's pre-game comprehension quiz.

Requires the plugin's reference-game-cued-fidelity branch for the two new options.
Adds reference-game-hawkins-local.html, the vendored offline twin, since the CDN
build cannot run until the plugin publishes to npm; vendor/ is gitignored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant