fix(metadata): re-vendor @jspsych/metadata from upstream main + sync tooling - #151
Merged
Conversation
fix: improve error logging and handling for OSF uploads
fix: fall back to valid PAT when OAuth token refresh fails
Upload queue retry with automatic recovery and dashboard UI
Merge test into main
docs: add FAQ entry about the 32 MB request size limit
…tooling
DataPipe's vendored @jspsych/metadata was a June-2024 fork frozen at v0.0.1
that silently discarded nested object/array trial data. The fix lives on the
upstream main branch but is NOT in the published npm 0.0.3 (which has the same
data-loss bug and would silently drop all data given DataPipe's pre-parsed
input). Rather than depend on the stale npm release or live-track a moving
branch, vendor a PINNED upstream commit and rebuild from it, with tooling to
make future re-syncs a one-command, reviewable step.
- functions/scripts/sync-metadata.mjs (+ npm run sync:metadata): clone upstream
at a ref, build packages/metadata, copy the built dist + sanitized
package.json + LICENSE into functions/metadata/, and record provenance in
VENDORED_FROM.json. Strips the package's scripts (upstream's
prepare:"npm run build" would break `npm install` of the file: dep, since we
ship dist-only) while keeping the csv-parse runtime dep.
- .github/workflows/metadata-drift-check.yml: weekly non-blocking job that
opens/updates a tracking issue when upstream main moves past the pinned commit.
- functions/metadata/: now dist-only, pinned to upstream main 224d336. dist is
committed (deploys need no metadata build); .gitignore updated to un-ignore it.
- functions/package.json: dep stays file:metadata; add explicit typescript
devDep (the build had relied on it transitively via the removed fork).
- functions/src/metadata-production.ts: generate()'s 3rd arg is now a string
ext ('json'|'csv'), not the old boolean csv flag.
- metadata-production.test.js: fixture updated to real output (type -> @type,
numeric -> number); data-derived levels/min-max double as a silent-drop guard;
fixed a pre-existing aliasing bug in the options test.
Verified: metadata-production, metadata-update, metadata-process suites pass;
functions build (tsc) and npm install are clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
functions/metadata is now a pre-built vendored dist with no lockfile or build script, so the "npm ci && npm run build" step in that directory fails with EUSAGE. The dist is committed and installed as a file: dependency by the existing functions npm ci step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both suites used logs/testlog and each deletes it at test start; jest runs them in parallel workers, so the base64 suite's delete could wipe the data suite's saveData counter between write and read (doc exists via the base64 increment, saveData undefined). Rename the base64 suite's doc to base64-testlog, matching its other doc IDs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DataPipe's vendored
@jspsych/metadataatfunctions/metadata/was a June-2024 fork ofv0.0.1, frozen since the standalone jspsych/metadata repo took over development. This PR re-vendors it from upstreammain(pinned to224d336) and adds tooling so future re-syncs are a deliberate one-command step instead of another multi-year drift.Why not just depend on the npm package?
Published
@jspsych/metadata@0.0.3predates the fixes we need (nested-data expansion, extraction API, array-acceptinggenerate()— landed in upstream PRs #115/#116/#117/#120 but never released). Worse,0.0.3'sgenerate()internallyJSON.parses its input and early-returns on failure, so a naive^0.0.3bump would make DataPipe (which pre-parses the data) silently emit only the default metadata template, dropping all real variables. Pinned re-vendor of upstream main is the safest path until upstream cuts a new release.What this fixes
responseobjects, mouse-tracking arrays) to contentless stubs. The re-vendored lib expands them into dotted sub-variables (response.Q0,mouse_tracking_data.x/y/t) with correct types/min/max."type"→"@type"onvariableMeasuredentries (verified safe: DataPipe's merge logic never reads.type)."number"typing (0.0.1 hardcoded"numeric"for system fields)."unknown"instead of accumulating comma-joined compound keys.getExtractedArrays()/getExtractedObjects()/getArrayJoinKeys()— needed for the follow-up Psych-DS layout work (feat(metadata): Psych-DS-compliant data layout on OSF + Firestore-only metadata transaction #152, stacked on this branch).Changes
functions/metadata/— now built dist only (pinned to upstream main224d336), withVENDORED_FROM.jsonprovenance + generated README; src/tests/build-configs removed.functions/scripts/sync-metadata.mjs(+npm run sync:metadata) — clones upstream at a given ref, buildspackages/metadata, copies dist + sanitizedpackage.json+ LICENSE, writes provenance. Sanitization strips upstream'spreparescript (would breaknpm installof a dist-onlyfile:dep) and devDeps, keeps thecsv-parseruntime dep..github/workflows/metadata-drift-check.yml— weekly non-blocking check; opens/updates a tracking issue when upstream main moves past the pinned commit..github/workflows/node.js.yml+ bothfirebase-deploy*.yml— removed the now-obsolete "npm ci && npm run buildinfunctions/metadata" step: the vendored directory is dist-only with no lockfile, so the step fails with EUSAGE (and there is nothing left to build).functions/src/__tests__/base64data-emulator.test.js— fixed a pre-existing CI race unrelated to the re-vendor but exposed by its CI runs: this suite anddata-emulator.test.jsboth used (and deleted) the samelogs/testlogdoc while jest runs suites in parallel, so one suite's cleanup could wipe the other'ssaveDatacounter mid-test. The base64 suite now uses its ownbase64-testlogdoc.functions/src/metadata-production.ts— adapts to the newgenerate()signature (3rd arg boolean →ext: 'json'|'csv').functions/src/__tests__/metadata-production.test.js— fixture updated to real new output; data-derived levels/min-max double as a silent-drop guard; fixed a pre-existing aliasing bug in the options test.functions/package.json— dep staysfile:metadata; added explicittypescriptdevDep (build previously relied on it transitively via the old fork)..gitignore— un-ignores the vendored dist.Testing
functionstscbuild clean; freshnpm installclean (no prepare-script breakage).build (22.x)passes on this branch (covers the emulator suites that need Java/firebase-tools).🤖 Generated with Claude Code