Wire DataPipe egress with per-round incremental saves (C&WG)

[htsukamoto5]

Closes #3.

Adds the save target this file has never had. Until now the only egress was a download button the participant had to choose to click, on a screen that also said "You may close this tab" — so every dropout, lobby timeout, and spectator contributed nothing. Those are precisely the sessions needed to characterise attrition.

Stacked on #14 (feat/prolific-identifiers-and-codes), not main. It builds on the CONFIG block and identifiers that PR adds rather than duplicating them — the save-policy values join that block, each annotated with its DECISIONS.md row. Review #14 first; this diff is only the egress layer and its tests.

C&WG only. Hawkins is untouched on purpose (A2).

The decisions this encodes are suggestions, not settled calls

• A4 — per-round chunks, nonce-suffixed filenames (SAVE_PER_ROUND)
• A3 — the abort path flushes unconditionally (FLUSH_ON_ABORT)
• B4 — assumes limitSessions is off. DataPipe's counter increments per save call, so per-round saving spends ~7 sessions per participant; a limit sized to participants would cap the run about a seventh of the way through and drop every later save silently.

Changing any of these is a one-line edit in CONFIG; nothing below hardcodes an equivalent literal.

The retry classifier is the part worth reviewing

Written against DataPipe's server source (jspsych/datapipe, functions/src/api-data.ts) rather than the plugin's example snippet, because the response semantics decide whether retrying is correct or actively harmful:

Response	Meaning	What we do
201	Uploaded to OSF	Success
202	OSF upload failed, but DataPipe persisted the data and queued its own server-side retry	Success. Retrying would duplicate rows
400 OSF_FILE_EXISTS	Filename collision. Not queued — the one path where data is genuinely dropped	Terminal (except the replay case below)
400 others	Session limit, validation, unknown experiment, PROVIDER_NOT_CONNECTED — configuration errors, not transient faults	Terminal; retrying burns the redirect budget and still loses the data
429, 5xx, network	Genuinely transient	Retry with backoff, honouring Retry-After

Classification keys on status class, not error strings. That is what makes it robust to the in-flight provider-migration branch upstream, which moves duplicate detection into a Firestore collision cache and makes OSF one provider among several — the client contract is unchanged, and any new 4xx is terminal by construction.

The subtle one

The nonce is fixed for the lifetime of a save, so all retries within one save post the same filename — that is what makes them idempotent. But if attempt 1 lands at OSF and only its response is lost to a network fault, attempt 2 gets OSF_FILE_EXISTS. Classifying that as terminal requeues the rows and re-sends them under a fresh nonce, duplicating them in the dataset — exactly what the nonce exists to prevent.

A collision from attempt 2 onward now counts as saved. A collision on the first attempt is a genuine clash with another session and stays terminal. This was found by writing the reassembly test, not by reading the code.

Other behaviour

• Chunks are disjoint and carry row_range, so reassembly is a concatenation and a duplicate chunk is detectable rather than silently merged.
• Failed chunks return their rows to the queue for a later flush. Duplicates are recoverable in analysis; missing rows are not.
• flush() races the save against a redirect budget, so no exit path can strand a participant waiting on OSF before they can submit (Prolific plumbing: completion codes and submission redirect #7). The submit button is wired before the flush resolves.
• pagehide + keepalive catches tab-close, back-navigation, and connection loss (A3). keepalive caps bodies at 64KB, which per-round chunks stay well under — another reason A4's chunking is load-bearing rather than merely tidy.
• Filename keys prefer PROLIFIC_PID, fall back to the adapter participant id locally, and dyad_id falls back to SEED so a filename never contains the literal null.

Wired exits

Per-round, completion, spectator, and pagehide. The two that don't exist yet are one Pipeline.flush(label) call each once their screens land: the partner-dropped abort (#5) and the no-match lobby exit (#6).

Testing

node tests/pipeline.test.mjs — 34 checks, no dependencies, no browser. The module is extracted from the experiment file and run against a fake DataPipe that distinguishes attempts from what actually lands at OSF, since a failed POST leaves no file behind.

Covers chunking and disjointness, every response class, requeue-on-failure, concurrent saves not double-sending, filename uniqueness and fallbacks, the redirect budget, idempotent replay, inert behaviour when unconfigured, and end-to-end reassembly of a partial lossy session.

Not browser-tested end to end — that needs a live experiment ID.

For the analysis side

Chunks reassemble by concatenating trials across a dyad's files; the two participants join on dyad_id.

Each chunk carries row_ranges: a list of inclusive [start, end] pairs, not a single pair. A chunk may carry rows reclaimed from an earlier failed save alongside its new ones, and those are not contiguous with each other. The invariant to check is that the union of all row_ranges for a participant covers 0..N exactly once — no gaps, no overlaps. chunk_seq may legitimately gap, since a save that fails outright still consumes a sequence number, so it is not a completeness check.

Before this does anything

CONFIG.DATAPIPE_EXPERIMENT_ID is empty, so saving is inert and warns to the console. Deliberate — safer than a placeholder that uploads to an unintended project. It needs #15, which is now the blocking step for seeing any of this work for real.

Known follow-ups

• dyad_id and SEED both read ?mp_session=. Firebase swap silently breaks SEED — every dyad would share one trial order #9 re-derives that from the Firebase room id; it should follow automatically, but verify it does — a dyad_id that differs between partners silently unjoins every dyad in the dataset.
• Chat transcripts are uploaded verbatim. D3 and D6 in DECISIONS.md are open IRB questions on retention and researcher review, and they now have a concrete destination to be answered about.

🤖 Generated with Claude Code

---

Review follow-ups (5ad35d0)

The blocking bug was real and is fixed. savedThrough -= rows.length assumed saves fail in LIFO order. Reproduced exactly as described: rows 0-1 lost permanently, 3-4 duplicated, ranges overlapping. The compounding problem was the silence — those overlapping ranges break the very contiguity invariant this PR told the analysis side to check, so the detection mechanism stops holding precisely when it is needed.

savedThrough is now a high-water mark that only advances. Failed saves push their [start, end] ranges onto a pending list, and each save claims everything pending plus everything new. That is why chunks now emit row_ranges (a list) rather than row_range (a pair) — a chunk may carry reclaimed rows that are not contiguous with its new ones. The analysis section above is updated accordingly.

The regression test is included. It is exactly the case the old concurrency test missed, because both of its saves succeeded.

• PID removed from filenames, payload only. Agreed on the reasoning: an OSF listing is browsable without opening a file, which is a broader exposure than the same value inside a row, and D3/D6 are open with IRB. Reconciliation reads the payload and is unaffected.
• classify() now accepts any 2xx. Agreed — treating an unexpected 200 as failure would requeue rows that landed and duplicate them.
• The completion screen no longer promises a retry that may not happen, distinguishing a redirect-budget timeout from a genuinely queued 202.
• configured() uses truthiness; the test's extraction regex now fails with a readable message.
• dyadKey() is documented as defensive-only, pointing at DYAD_ID's own fallback added on the base branch in c2c69c2.

Tests: 39, up from 34.

[htsukamoto5]

Marking as draft: this overlaps PR #14, which already closes #4 with an equivalent CONFIG block, identifiers, run provenance, and viewport capture. Restructuring so this carries only the #3 egress work, stacked on #14, rather than duplicating it.