Storage provider migration: provider interface, Google Drive adapter, and provider-aware UI - #154
Merged
Conversation
Captures the OSF dependency audit, provider evaluation (Box/S3/Dropbox/ OneDrive, GitHub, Dataverse ecosystem/ICPSR/Dryad/Databrary), and the pluggable-provider architecture targeting Google Drive, Figshare, and Dataverse, with OSF retained as a legacy adapter. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Make Figshare/Dataverse selection conditional on gating spikes, with Box as the pre-approved substitute - Scope base64/media support to all providers at launch; define WriteResult and size/quota capability hints - Add claim lifecycle (pending/confirmed) and rehydration locking to the collision cache; dual-run against OSF 409 for validation - Document auth-longevity constraints (Google verification lead time, Dataverse token expiry) and add QUOTA_EXCEEDED to the error taxonomy - Add testing strategy, build step 0 (OAuth app registration), and named disqualification criteria per spike Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ypes Build step 1 of the provider migration (docs/provider-migration-design.md): - StorageProvider interface with WriteResult union, generic error taxonomy, and capability hints - provider registry (register/get/list, test-only reset) - additive optional fields on ExperimentData and UserData; legacy OSF fields unchanged - promote typescript to a direct devDependency of functions/ (was only available transitively via @jspsych/metadata) No behavior change; nothing consumes the interface yet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build step 2 of the provider migration: pure refactor, no behavior change. - OSF adapter (providers/osf.ts) implementing StorageProvider by delegating to the existing put-file-osf/update-file-osf modules; maps OSF statuses into the generic error taxonomy - getProviderForExperiment() resolves the adapter + container ref, with legacy experiments (no storageProvider field) defaulting to OSF - api-data, api-base64, metadata-block, scheduled-upload-retry switched to the interface; response codes, queue fields, and log strings unchanged - preserves the latent 210 status check in metadata-block verbatim (removed in build step 3 per the design doc) - 6 new adapter unit tests; full emulator suite green (18 suites, 114 tests) with no existing test modified Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build step 3a of the provider migration. Duplicate-filename detection now
lives in Firestore, decoupled from the provider:
- collision-cache.ts: salted-hash claims (raw filenames never stored) at
experiments/{id}/filenameClaims/{hash} with a pending->confirmed
lifecycle, owner tokens for idempotent retry re-entry, stale-pending
takeover (15 min), 90-day TTL via expiresAt, and lazy rehydration from
provider.listFiles behind a 60s lease that fails loudly if listing fails
- api-data/api-base64 claim before every provider write; duplicates are
rejected without a provider round-trip; OSF's 409 stays as a dual-run
backstop with disagreements logged (collisionCacheDisagreement)
- retry queue carries claimToken; retries re-enter their own pending claim;
legacy queue docs without a token skip the cache
- confirmClaim skips the warmUntil bump when it's already >89 days out,
avoiding a second experiment-doc write per submission under burst load
- early-persist test's inline mock gains a GET route (rehydration lists
the container on first write); its assertions are unchanged
TDD: 12 unit + 5 integration tests written and reviewed red before
implementation. Full emulator suite green.
Deploy note: the Firestore TTL policy on filenameClaims.expiresAt must be
enabled via console/gcloud when this ships; the emulator does not enforce
TTL and nothing depends on it for correctness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… write
Build step 3b of the provider migration; closes out build step 3.
- putFileOSF parses the 201 response body; WriteResult.fileRef now carries
the provider file id
- metadata-block reads/stores metadataFileRef on metadata/{id} instead of
listing the provider folder per submission. Three-state semantics:
undefined = pre-migration doc, triggers one-time legacy discovery via
listFiles; null = known absent, create without listing; ref = update
directly. Stale refs self-heal (recreate + store new ref). Refs without
a usable id are not stored, degrading to re-discovery instead of
persisting an un-updatable ref
- the latent 210 status check is removed: the firestore-has-metadata /
provider-doesn't branch now succeeds instead of always throwing (the old
emulator test masked this by asserting only metadataMessage, which was
populated before the throw)
- metadata-process.ts deleted (no remaining callers) along with its unit
test; metadata-emulator's runTransaction block rewritten from the
token-flip mock trick to direct structural seeding of the four
firestore x provider metadata scenarios
TDD: 9 unit/integration contract tests written and reviewed red first.
Full emulator suite green (21 suites, 142 tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ite path Build step 4a of the provider migration — the first non-OSF provider, proving the abstraction end-to-end for writes, collision cache, metadata, and the retry queue. - providers/gdrive.ts: app-created DataPipe root folder + per-experiment subfolders, hand-built multipart uploads, media-PATCH updates, paginated listFiles (fails loudly on listing errors — Drive has no 409 backstop, so rehydration completeness is load-bearing), alt=media downloads, and reason-based Drive error mapping (never NAME_CONFLICT: the collision cache is the only duplicate gate) - StorageProvider gains downloadFile; OSF adapter implements it and metadata-download.ts is deleted - resolveToken dispatches on storageProvider: gdrive path reads connectedAccounts.gdrive, refreshes via GDRIVE_TOKEN_URL with rotation, persists tokens encrypted; PROVIDER_NOT_CONNECTED added to api-messages - metadata-block fully provider-generalized (shared resolveToken replaces its duplicated inline OSF token logic); self-heal handles both provider failure styles but refuses to self-heal on AUTH_EXPIRED / RATE_LIMITED / QUOTA_EXCEEDED - queue docs carry storageProvider/providerContainer (legacy OSF fallback preserved); scheduled-upload-retry parameterized accordingly - GDRIVE_API_BASE/GDRIVE_TOKEN_URL env overrides (call-time reads); functions/.env.datapipe-test wires the emulator to the tests' mock Drive server TDD: 17 unit + 5 integration contract tests reviewed red first. Full emulator suite green (24 suites, 174 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The collisionCacheDisagreement audit entry was written after the 400 response was already sent, so observers of the log (including the step-3a integration test under parallel-suite load) raced the write. Logs in the disagreement branch now land before the response in both apidata and apibase64 — same fix class as the earlier get-condition log race. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… flow Build step 4b, completing build step 4 of the provider migration. The OSF OAuth flow is an identity flow (signup/sign-in/linking with Firebase custom tokens) and stays byte-identical. New providers get a separate storage-grant flow for already-authenticated users: - providers/oauth-config.ts: per-provider OAuth registry (gdrive only for now; env-driven, call-time reads; figshare becomes a config addition) - generateOAuthState optionally takes a provider: state doc records it and the response includes a ready-made authorize URL with access_type=offline&prompt=consent (without which Google never issues a refresh token) - connectProvider: CSRF-state validation (single-use, provider-matched), verifyIdToken uid ownership, code exchange, hard-fail if the exchange returns no refresh_token (no half-connected accounts), sibling-safe encrypted persistence into connectedAccounts.<provider> - disconnectProvider: same auth, FieldValue.delete of the map entry - gdrive token refresh extracted to providers/gdrive-oauth.ts, shared by resolveToken and a new scheduled pass (refreshExpiringGdriveTokens, 10-min window) that cannot break the OSF pass - design doc: documents the pre-existing test hazard that the OSF refresh path has no URL override and can hit production accounts.osf.io from emulator tests TDD: 17 contract tests reviewed red first; refresh-helper tests use per-token assertions and a discriminating fetch mock so the helper's collection-wide scan stays isolated from parallel suites. Full emulator suite green twice consecutively (27 suites, 191 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build step 7a. OSF experiment creation stays browser-driven and untouched; non-OSF providers get a server-side endpoint since their container creation (createDataContainer) is server-only. - create-experiment.ts: verifies idToken ownership, resolves the provider token, creates the storage container, and batch-writes the experiment doc + users.experiments arrayUnion with exact parity to the client's id format (12-char nanoid alphabet) and default fields (including the requiredFields ["trial_type"] default the client hardcodes); rejects osf/unknown providers - firestore.rules: verifyFields split into baseFields + a conditional hasAll — provider-shaped docs require storageProvider+providerContainer, legacy docs still require the OSF trio; users rules (and the clients-cannot-write-connectedAccounts guarantee) unchanged and pinned by a new rules test - scheduled-pending-recovery: promoteToQueue now passes storageProvider/providerContainer through with omit-if-undefined — fixes a real bug where recovery of a gdrive experiment's pending upload threw on undefined osfFilesLink and was silently swallowed - /api/createexperiment rewrite; verifyOwnership shared from connect-provider; nanoid added to functions deps; gdrive-emulator's mock server gained the same EADDRINUSE retry its port-3579 sibling uses TDD: 10 contract cases reviewed red first (rules gap, endpoint 404s, and the recovery bug all demonstrated before implementation). Full emulator suite green twice consecutively (29 suites, 205 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build step 7b, completing the frontend (design-doc step 7).
- lib/provider-config.js: STORAGE_PROVIDERS map — new providers become
config entries (name, isConnected, container link/label); OSF keeps its
bespoke legacy UI deliberately
- admin/new.js: storage selector (native radio inputs — Chakra's compound
RadioGroup defers state outside act(), breaking synchronous tests; OSF
path byte-identical and pinned by regression test). gdrive branch:
connect CTA when unlinked, title-only form posting to
/api/createexperiment via createProviderExperiment, routes to the new
experiment on success
- ProviderConnections + account.js Storage Providers section: connect via
generateOAuthState {provider} -> authorizeUrl redirect; disconnect via
/api/disconnectprovider; status reads connectedAccounts reactively
- pages/oauth2/connect.js: storage-grant OAuth callback (CSRF-checked,
sign-in required, posts to /api/connectprovider) — separate from the
OSF identity callback by design
- ExperimentInfo renders the provider container link for provider-backed
experiments; legacy OSF rows unchanged. QueuePanel copy generalized to
provider-neutral phrasing
- firebase.json: /api/connectprovider + /api/disconnectprovider rewrites
TDD: 14 RTL contract cases reviewed red first (3 as pinned regression
guards). Next.js production build clean; full emulator suite green twice
(35 suites, 223 tests, --maxWorkers=2).
Deploy note: GDRIVE_REDIRECT_URI must point at /oauth2/connect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- design doc gains a consolidated deployment checklist (OAuth app verification, functions env vars, TTL policy, refresh-query index, FAQ pointer, deploy order) - CI runs jest with --maxWorkers=2: the emulator-backed suites contend under full parallel load (pre-existing data-emulator timing flake) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR migrates DataPipe’s storage layer from OSF-only writes to a pluggable storage-provider architecture, adding Google Drive support end-to-end (backend provider interface + adapters, OAuth connect, server-side experiment creation, collision handling, and provider-aware UI).
Changes:
- Introduces a storage provider abstraction/registry in Functions, with OSF and Google Drive adapters, and routes uploads/metadata through the provider layer.
- Adds Firestore-backed filename collision cache (claim/confirm lifecycle) and threads provider/container metadata through queueing + scheduled retry/recovery paths.
- Adds Google Drive OAuth connect/disconnect flow, server-side experiment creation for non-OSF providers, and provider-aware UI (account connections, experiment creation, dashboard panels), plus new/updated emulator + unit tests and CI worker capping.
Reviewed changes
Copilot reviewed 63 out of 65 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pages/oauth2/connect.js | New provider OAuth callback page for connect flows (non-OSF). |
| pages/admin/new.js | Provider selector + Google Drive experiment creation path in UI. |
| pages/admin/account.js | Adds “Storage Providers” section on account page. |
| components/account/ProviderConnections.js | UI for connecting/disconnecting provider accounts via OAuth. |
| lib/provider-config.js | Frontend registry for non-OSF storage providers (gdrive). |
| lib/experiment-creation.js | Adds createProviderExperiment() helper calling /api/createexperiment. |
| components/dashboard/ExperimentInfo.js | Provider-aware container link rendering for experiment dashboard. |
| components/dashboard/QueuePanel.js | Provider-neutral copy for queued upload UI. |
| functions/src/providers/types.ts | Defines provider abstraction types + connected account schema types. |
| functions/src/providers/registry.ts | Provider registration/lookup utilities. |
| functions/src/providers/index.ts | Registers OSF + gdrive providers and adds getProviderForExperiment(). |
| functions/src/providers/osf.ts | OSF adapter implementing provider interface (write/update/list/download). |
| functions/src/providers/gdrive.ts | Google Drive adapter implementing provider interface. |
| functions/src/providers/oauth-config.ts | Provider OAuth config registry (gdrive) read at call time. |
| functions/src/providers/gdrive-oauth.ts | Shared gdrive refresh-token logic for lazy + scheduled refresh. |
| functions/src/connect-provider.ts | New connect/disconnect provider endpoints with server-side CSRF + ownership verification. |
| functions/src/create-experiment.ts | New server-side experiment creation endpoint for non-OSF providers. |
| functions/src/generate-oauth-state.ts | Adds optional provider param + authorizeUrl generation for provider OAuth. |
| functions/src/resolve-token.ts | Adds provider-aware token resolution (OSF vs gdrive). |
| functions/src/api-data.ts | Routes writes via provider + collision-cache claim/confirm; queues on failures. |
| functions/src/api-base64.ts | Same as api-data for base64 uploads via provider + collision cache. |
| functions/src/collision-cache.ts | New Firestore-backed collision cache (salt + filename claim lifecycle). |
| functions/src/queue-upload.ts | Queues uploads with additive provider fields + claimToken, omitting undefined Firestore fields. |
| functions/src/scheduled-upload-retry.ts | Scheduled retry generalized to provider writes + collision cache semantics. |
| functions/src/scheduled-pending-recovery.ts | Preserves provider fields when recovering stale pending uploads into the queue. |
| functions/src/metadata-block.ts | Provider-aware metadata update path with metadata file ref tracking + self-heal behavior. |
| functions/src/metadata-process.ts | Removed legacy OSF-only metadata discovery (replaced by provider ref tracking). |
| functions/src/metadata-download.ts | Removed legacy OSF-only metadata download (replaced by provider download). |
| functions/src/put-file-osf.ts | Surfaces fileId/fileName from OSF upload response to support ref tracking. |
| functions/src/interfaces.ts | Adds provider-migration fields/types to Experiment/User/Queue interfaces. |
| functions/src/scheduled-token-refresh.ts | Adds scheduled proactive gdrive token refresh pass after OSF refresh pass. |
| functions/src/api-messages.ts | Adds PROVIDER_NOT_CONNECTED message. |
| functions/src/index.ts | Exposes new Functions endpoints (connect/disconnect/createexperiment). |
| functions/package.json | Adds nanoid + typescript dependency updates for Functions. |
| functions/package-lock.json | Lockfile updates for Functions dependencies. |
| functions/.env.datapipe-test | Adds gdrive env vars for emulator/testing (API base, token/authorize URLs, redirect URI, encryption key). |
| firestore.rules | Updates experiment rules to accept either OSF fields or provider fields. |
| firebase.json | Adds rewrites for new API endpoints (createexperiment/connectprovider/disconnectprovider). |
| .github/workflows/node.js.yml | Caps Jest workers for emulator-backed CI stability. |
| package-lock.json | Root lockfile changes (dependency metadata updates). |
| functions/src/tests/resolve-token-gdrive.test.js | Unit tests for gdrive token resolution + refresh behavior. |
| functions/src/tests/put-file-osf-ref.test.js | Unit tests for OSF upload response parsing + osfProvider surfacing. |
| functions/src/tests/providers-registry.test.js | Unit tests for provider registry behaviors. |
| functions/src/tests/providers-osf.test.js | Unit tests for osfProvider mapping/listing behavior. |
| functions/src/tests/pending-recovery-provider-regression.test.js | Emulator regression test ensuring recovery carries provider fields. |
| functions/src/tests/oauth-connect-scheduled-regression.test.js | Regression guard for OSF scheduled refresh behavior. |
| functions/src/tests/oauth-connect-refresh-emulator.test.js | Emulator test for proactive gdrive refresh helper. |
| functions/src/tests/metadata-ref-emulator.test.js | Emulator integration tests for metadata ref tracking + self-heal. |
| functions/src/tests/metadata-emulator.test.js | Rewritten metadata emulator matrix tests for ref-based behavior. |
| functions/src/tests/early-persist-emulator.test.js | Mock OSF server updated to support collision-cache listing on cold start. |
| functions/src/tests/collision-integration-emulator.test.js | Emulator integration tests for collision cache behavior and queueing. |
| tests/queue-panel.test.jsx | UI test pinning provider-neutral QueuePanel copy. |
| tests/provider-connections.test.jsx | UI tests for connect/disconnect interactions and CSRF/localStorage behavior. |
| tests/provider-config.test.js | Tests for STORAGE_PROVIDERS.gdrive config/behavior. |
| tests/new-experiment-page.test.jsx | UI tests for provider selector and gdrive experiment creation path. |
| tests/experiment-info.test.jsx | UI tests for provider-aware ExperimentInfo rendering. |
| tests/connect-callback-page.test.jsx | UI tests for provider OAuth callback page behavior. |
| tests/firestore-rules.test.js | Adds tests for provider-migration generalization in Firestore rules. |
Files not reviewed (1)
- functions/package-lock.json: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
24
to
32
| if (reason.includes("OSF error 503") || reason.includes("OSF error 502")) { | ||
| return "OSF was temporarily unavailable."; | ||
| return "Your storage provider was temporarily unavailable."; | ||
| } | ||
| if (reason.includes("OSF error 429")) { | ||
| return "OSF rate-limited the request."; | ||
| return "Your storage provider rate-limited the request."; | ||
| } | ||
| if (reason.includes("OSF error 401") || reason.includes("OSF error 403")) { | ||
| return "Authentication error. Your OSF token may need to be refreshed."; | ||
| return "Authentication error. Your storage provider connection may need to be refreshed."; | ||
| } |
- api-data/api-base64 record queue failures as "Provider error <status>" (they can come from any provider now); QueuePanel maps both the new prefix and the legacy "OSF error" strings still stored in queue docs - gdrive writeSessionFile builds its FileRef from the already-fallbacked storedFilename, restoring the FileRef.name contract without the cast - test-env GDRIVE_REDIRECT_URI points at /oauth2/connect, the route that actually exists Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ProviderConnections: use relative /api/generateoauthstate path,
matching the rest of the new provider code's rewrite convention
- ExperimentInfo: render per-provider containerLinkText ("Open folder")
instead of the raw Drive folder ID as the container link text
- api-data/api-base64: cleanupPending in both duplicate 400 branches
so duplicate submissions don't take a lap through pending recovery
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reconciles the provider-migration write-path/collision-cache work with the
Psych-DS metadata pipeline that landed on test in parallel. The two features
touched the same files; test wired Psych-DS entirely to OSF (putFileOSF,
osfFilesLink, subfolder.ts). This merge generalizes that pipeline through the
StorageProvider interface so metadata (raw file under data/raw/, derived
sidecar CSVs under data/, .psychds-ignore) works on ANY provider, gdrive
included.
Key reconciliations:
- metadata-block.ts: unions both rewrites — provider-generalized
dataset_description.json handling (metadataFileRef three-state tracking,
four-state message, self-heal excluding AUTH_EXPIRED/RATE_LIMITED/
QUOTA_EXCEEDED) plus Psych-DS derived-file production and the
discriminated-union return that carries derivedFiles only on success. Takes
the already-resolved provider token from api-data.
- metadata-derived-upload.ts: routes derived-file uploads and queueing through
provider.writeSessionFile; DerivedUploadTarget carries storageProvider/
providerContainer (legacy osfFilesLink fallback), resolved the same way
scheduled-upload-retry does. Drops the OSF-only "resolve data/ once"
batch optimization — each file walks its own path through the provider.
- providers/gdrive.ts: writeSessionFile walks all path segments (nested
Psych-DS folders); listFiles BFS-recurses (leaf names, fail-loud at every
level, paginated) so collision-cache rehydration finds raw files now stored
under data/raw/ — gdrive has no 409 backstop, so listing completeness is
load-bearing.
- api-data.ts: claim on the raw leaf filename, write to uploadPathFor()'s
data/raw/ path, upload/queue derived files via the provider-aware target.
Adopts test's keep-pending-on-metadata-failure so scheduled-pending-recovery
can salvage the raw data instead of dropping it.
- put-file-osf.ts / subfolder.ts: test's arbitrary-depth path walking plus
provider-migration's 201-body fileId/fileName parsing; node-fetch import
convention extended to subfolder.ts so module-level test mocks route through
it.
Test-fixture updates (behavior intentionally changed, coverage preserved):
- metadata-emulator / metadata-ref-emulator mock servers gain a kind=folder
route (WaterButler links.move) so data/raw/ resolves; getListingCallCount
expectation reflects per-file folder resolution, not metadata re-discovery.
- put-file-osf / metadata-derived-upload emulator tests switch to module-level
jest.mock("node-fetch"); one batching-count assertion updated for the
dropped optimization.
- providers-gdrive listFiles test expects the new recursive listing.
Full emulator suite green: 37 suites, 240 tests. tsc clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates DataPipe's storage layer from OSF-only writes to a pluggable storage-provider architecture, with Google Drive as the first new provider.
Test plan
npm run test-ci(root) and functions test suite against emulators (--project datapipe-test)🤖 Generated with Claude Code