feat(metadata): Psych-DS-compliant data layout on OSF + Firestore-only metadata transaction

[htsukamoto5]

⚠️ Stacked on #151

This branch includes the three commits of #151 (fix/metadata-version-drift) — review only the last five commits here, or view the diff against the #151 branch. Base is test so CI runs; merge #151 first, then this.

Part 1: Firestore-only metadata transaction (refactor)

blockMetadata's entire body — including every OSF network call — ran inside db.runTransaction. Firestore retries the transaction callback on contention (e.g. two participants submitting concurrently), which would re-run OSF uploads: a latent duplicate-write risk. The transaction is now strictly read → merge → write on the Firestore metadata doc; OSF I/O happens before it (existence check) and after it (mirror upload). When the merge needs the OSF copy as its base (Firestore empty, OSF populated — first submission after a wipe), the transaction aborts via a sentinel, the copy is downloaded outside it, and the transaction re-runs.

Fixes three pre-existing bugs found in the process:

• The Firestore-only branch checked putFileOSF's result with errorCode !== 210, which throws even on success (success returns errorCode: null), so that branch could never complete.
• The OSF-only branch uploaded the unmerged incoming metadata to OSF while Firestore got the merged version; both now receive the merged one.
• The create branch ignored putFileOSF's result entirely (silent metadata loss); it is now handled (see resilience model below).

blockMetadata also now receives the token api-data already resolved via resolveToken, instead of re-deriving it with ~25 lines of duplicated logic that could trigger a second token refresh per submission.

Part 2: Psych-DS-compliant data layout on OSF

With the metadata toggle on, DataPipe now ships a genuinely Psych-DS-compliant dataset to OSF instead of a lone raw file next to dataset_description.json. A JSON submission abc123.json with a survey response object and a mouse_tracking_data array produces:

<OSF component root>/
  dataset_description.json
  .psychds-ignore
  data/
    subject-abc123_data.csv                              ← main data table
    subject-abc123_measure-response_data.csv             ← object-column sidecar
    subject-abc123_measure-mouseTrackingData_data.csv    ← array-column sidecar
    raw/
      abc123.json                                        ← original, byte-for-byte

With the toggle off, nothing changes: the raw file goes to the component root as today.

How it works

• The raw file is the critical upload. The participant's submission lands byte-verbatim at data/raw/<original name> and is what session counting and queue-on-failure key off. Everything else is derived from it and uploaded best-effort after it lands — a failure of any derived file is queued in the existing uploadQueue (whose retry worker treats 409 as done) and logged, but never fails the submission and can never precede its source data.
• Conversion is the library's, not ours. The main CSV and sidecars come from one call to buildPsychDSDataFiles — the same shared function the metadata CLI and browser flows use — so DataPipe's filenames and CSV bytes match the CLI's for identical data, with zero conversion code to keep in sync across re-vendors. Since the CLI writes these per source file, per-submission output is the exact incremental equivalent: no cross-session accumulation or dedup state.
• CSV submissions stay byte-verbatim. The original CSV text is passed through as mainContent, so the main data file keeps its exact bytes (column order, quoting) instead of being re-serialised.
• JSON parsing is at parity with the CLI. produceMetadata now routes JSON through the library's parseJsonData, so a nonstandard { "trials": [...] } wrapper is unwrapped exactly as the CLI does; bare arrays (the standard jsPsych payload) are untouched.
• dataset_description.json's OSF mirror is now non-fatal. Firestore is the source of truth and every submission re-merges and re-mirrors, so a create failure queues for retry (409 = a concurrent submission won the create race — fine), an update failure self-heals on the next submission, and neither rejects the participant's data. Previously a create failure 400'd the whole submission.
• putFileOSF walks arbitrary-depth paths. WaterButler has no atomic deep-path create, so each folder level of data/raw/ is found-or-created by chaining folder move links (subfolder.ts), with a 409 on folder creation treated as "another submission created it first — re-list and continue" (this race is now hit on every metadata submission). Also switched from node-fetch to Node 22's global fetch.
• Researcher subfolders are flattened (condition-A/abc.json → data/raw/abc.json), matching how the CLI converts whole directories into a flat data/. Grouping by subfolder is lost under the Psych-DS layout; with metadata off, subfolders behave as before.
• Modules: metadata-sidecars.ts is deleted in favor of metadata-derived-files.ts (pure builder: full derived set + rawDataPath) and metadata-derived-upload.ts (best-effort upload/queue).

For reviewers

• Filename bases come from the library's deriveFallbackBase (subject-<stem>, an official Psych-DS keyword); sidecar names from its deriveArrayFilename/disambiguation. Cheap to change before merge if you want a different convention.
• Deliberately kept pre-existing behavior: if the metadata block itself hard-fails (metadata generation throws, or an OSF read fails), the submission still returns 400 and the raw file never uploads. Making that non-fatal would change the participant-facing API contract — flagged as a product decision, not changed here.

Testing

• 40 unit tests across 5 suites pass locally (tsc clean): metadata-derived-files.test.js (layout, naming, flattening, byte-verbatim CSV, disambiguation), put-file-osf.test.js (10 tests: depth 0/1/2 walks, folder create, 409 race, upload failures), plus the extended metadata-production (wrapped-trials parity, mainContent) and existing metadata-process/metadata-update suites.
• CI build (22.x) green, covering the emulator suites (each metadata-state branch's response message is preserved by the refactor).
• Live-verified against a real OSF project (throwaway project, token since revoked), driving the actual compiled modules: nested-folder create and find via WaterButler move links, 409 on duplicate folder create (the race branch), 409 on duplicate file, Buffer bodies under global fetch, the full layout above landing exactly as designed, and the raw file round-tripping byte-identical.

🤖 Generated with Claude Code