From 3ce25e8a96892738cd5bafaf75f850d0d1ae39ae Mon Sep 17 00:00:00 2001 From: Ruge Li Date: Wed, 7 Jan 2026 10:46:04 -0800 Subject: [PATCH 1/9] first draft --- CONTRIBUTING.md | 25 +++++++++++++++++++++++++ FIREBASE_SCHEMA.md | 19 +++++++++++++++++++ README.md | 3 +++ 3 files changed, 47 insertions(+) create mode 100644 FIREBASE_SCHEMA.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9bad934d..e059ef7f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,6 +17,31 @@ project documentation. If you cannot find the documentation you're looking for, please file a GitHub issue with details of what you'd like to see documented. +## Firebase Overview + +cellPACK Studio reads recipe data from Firebase and writes user edits back. The backend handles uploading recipes and running packing jobs. + +### Collections Used by cellPACK Studio + +| Collection | What It Does | Access | +|------------|--------------|---------------| +| `example_packings` | Maps recipes to UI display names and editable fields | Read | +| `editable_fields` | Defines which recipe fields users can edit | Read | +| `recipes` | Full recipe data with references | Read | +| `objects` | Object definitions (molecules, organelles) | Read | +| `gradients` | Gradient definitions for spatial distributions | Read | +| `composition` | Composition definitions | Read | +| `recipes_edited` | User-modified recipes | Write | +| `job_status` | Packing job progress | Poll (read) | + +> **Note:** Collections like `configs` and `results` are managed by the backend. For the complete database schema, see [FIREBASE_SCHEMA.md](FIREBASE_SCHEMA.md). + +### Getting Access + +- **Development:** Create your own Firebase project in test mode. See [Firebase Firestore tutorial](https://firebase.google.com/docs/firestore). +- **Staging:** Contact the code owner for credentials, then configure your `.env` file (see README). + + ## How to Contribute Typical steps to contribute: diff --git a/FIREBASE_SCHEMA.md b/FIREBASE_SCHEMA.md new file mode 100644 index 00000000..2a147f7e --- /dev/null +++ b/FIREBASE_SCHEMA.md @@ -0,0 +1,19 @@ +# Firebase Schema Documentation + +This document describes the Firestore database schema used by cellPACK. Collections are accessed by the client (cellPACK Studio), the core package(backend), or both. + +## Collections + +| Collection Name | Purpose | Key Fields | Read | Write | Cleanup Policy | Related Process | Notes | +| -------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------- | ---------------------------------------------------------------- | ----------------------------------------------- | +| **recipes** | Stores the full-structured recipes | `name`, `id`, `dedup_hash`, `description`, `version`, `format_version`, `bounding_box`, `composition`, `objects`, `gradients`, `optional_gradients` | Recipe loading, UI display, user edit comparison, packing execution | Backend uploads via admin CLI tool | **No cleanup** - Permanent storage | Recipe loading, reference resolution, comparison with user edits | `optional_gradients` manually added | +| **recipes_edited** | Stores user-modified recipes temporarily | Same as `recipes` + `recipe_path`, `timestamp` | Backend retrieval for packing | User submits modified recipe (differs from default) | **No automated cleanup** - Manual only | User recipe submission, packing job submission | Unnested format, no reference resolution needed | +| **configs** | Stores packing configuration settings | Referenced by path (e.g., `firebase:configs/{id}`) | Building packing job request, backend reads during packing | Backend admin uploads via CLI tool | **No cleanup** - Permanent storage | Packing job configuration, submitted to Docker server | Referenced but not directly queried by frontend | +| **objects** | Stores object definitions (molecules, organelles) | `name`, `id`, `dedup_hash`, `type`, `color`, `packing_mode`, `place_method`, `radius`, `inherit`, `gradient`, `representations`, `jitter_attempts` | Recipe reference resolution, inheritance chain resolution, packing execution | Backend admin uploads with deduplication | **No cleanup** - Permanent storage | Reference resolution in recipes, inheritance chain resolution | Handles inheritance chains | +| **gradients** | Stores gradient definitions for spatial distributions | `name`, `id`, `dedup_hash`, `description`, `mode`, `mode_settings`, `weight_mode`, `pick_mode`, `reversed`, `invert` | Recipe/object reference resolution, editable field options | Backend admin uploads with deduplication | **No cleanup** - Permanent storage | Reference resolution, gradient options for editable fields | Can be combined (multiple per object) | +| **composition** | Stores composition definitions (what objects go where) | `name`, `id`, `dedup_hash`, `count`, `molarity`, `object`, `inherit`, `regions` (interior, surface, leaflets) | Recipe reference resolution, region-based placement | Backend admin uploads with topological sort | **No cleanup** - Permanent storage | Reference resolution, region-based object placement | Handles nested composition dependencies | +| **job_status** | Tracks AWS packing job status | `status`, `error_message`, `outputs_directory`, `result_path`, `timestamp` | Frontend polling (every 500ms), progress monitoring, result retrieval | Backend updates during job execution (RUNNING → DONE/FAILED) | **No automated cleanup** - Manual only | Job monitoring, status polling, result retrieval | Not in backend default collection list | +| **example_packings** | Maps example recipes to configs and defines UI metadata | `name` (display name), `recipe` (recipe ID), `config` (config ID), `editable_fields` (array of field IDs), `result_path` | App startup, recipe dropdown population | Manual admin setup (for frontend only) | **No cleanup** - Permanent storage | App initialization, recipe dropdown population, UI configuration | **Frontend only** - Not used by backend | +| **editable_fields** | Defines which recipe fields users can edit in UI | `id`, `name`, `data_type`, `input_type`, `description`, `path`, `min`, `max`, `options`, `gradient_options`, `conversion_factor`, `unit` | App initialization, form generation, input validation | Manual admin setup (for frontend only) | **No cleanup** - Permanent storage | Dynamic form generation, input validation, gradient options | **Frontend only** - Not used by backend | +| **results** | Stores Simularium result metadata for auto-opening (backend only) | `batch_job_id`, `timestamp`, `url` (Simularium result S3 URL), `user` | Backend cleanup validation (S3 URL check) | Successful packing completion with Simularium output | **180 days** - Cleaned if S3 URL inaccessible | Result archiving, user history tracking, result URL storage | Cleanup validates S3 existence | + diff --git a/README.md b/README.md index cd4aeb5a..61a4de0d 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,9 @@ This client interacts with the cellPACK server, which consists of a variety of b * **ECR**: Docker image built from the [cellPACK github repo](https://github.com/mesoscope/cellpack) is published to the `cellpack-private` ECR repository. That image defines the container specificationsin which the batch job will run. * **CloudWatch**: Logs from each AWS Batch job are written to CloudWatch. These logs can be accessed via the GET /logs endpoint. +### cellPACK Database +* **Firebase Firestore**: The cellPACK database is hosted in Firebase Firestore. This database stores all recipes, objects, gradients, compositions, packing configurations, job statuses, and results metadata. See [CONTRIBUTING.md](CONTRIBUTING.md#firebase-overview) for Firebase overview and [FIREBASE_SCHEMA.md](FIREBASE_SCHEMA.md) for the complete database schema. + #### Resources * [Server Architecture Overview Diagram](https://docs.google.com/presentation/d/1eG2XCxgYNaoDIYI-M6Tzef17bGuFirZZhmfZlBFTaXc/edit#slide=id.g26c8fd413da_0_34) * [AWS Batch Dashboard](https://us-west-2.console.aws.amazon.com/batch/home?region=us-west-2#) From 2e14cd686d7e0f3e09de7b8e8b6c096912160675 Mon Sep 17 00:00:00 2001 From: rugeli Date: Thu, 8 Jan 2026 13:51:50 -0800 Subject: [PATCH 2/9] update --- FIREBASE_SCHEMA.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/FIREBASE_SCHEMA.md b/FIREBASE_SCHEMA.md index 2a147f7e..ec703442 100644 --- a/FIREBASE_SCHEMA.md +++ b/FIREBASE_SCHEMA.md @@ -7,13 +7,13 @@ This document describes the Firestore database schema used by cellPACK. Collecti | Collection Name | Purpose | Key Fields | Read | Write | Cleanup Policy | Related Process | Notes | | -------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------- | ---------------------------------------------------------------- | ----------------------------------------------- | | **recipes** | Stores the full-structured recipes | `name`, `id`, `dedup_hash`, `description`, `version`, `format_version`, `bounding_box`, `composition`, `objects`, `gradients`, `optional_gradients` | Recipe loading, UI display, user edit comparison, packing execution | Backend uploads via admin CLI tool | **No cleanup** - Permanent storage | Recipe loading, reference resolution, comparison with user edits | `optional_gradients` manually added | -| **recipes_edited** | Stores user-modified recipes temporarily | Same as `recipes` + `recipe_path`, `timestamp` | Backend retrieval for packing | User submits modified recipe (differs from default) | **No automated cleanup** - Manual only | User recipe submission, packing job submission | Unnested format, no reference resolution needed | +| **recipes_edited** | Stores user-modified recipes temporarily | Same as `recipes` + `recipe_path`, `timestamp` | Backend retrieval for packing | User submits modified recipe (differs from default) | **24 hours** - cleaned if older than 24 hours | User recipe submission, packing job submission | Unnested format, no reference resolution needed | | **configs** | Stores packing configuration settings | Referenced by path (e.g., `firebase:configs/{id}`) | Building packing job request, backend reads during packing | Backend admin uploads via CLI tool | **No cleanup** - Permanent storage | Packing job configuration, submitted to Docker server | Referenced but not directly queried by frontend | -| **objects** | Stores object definitions (molecules, organelles) | `name`, `id`, `dedup_hash`, `type`, `color`, `packing_mode`, `place_method`, `radius`, `inherit`, `gradient`, `representations`, `jitter_attempts` | Recipe reference resolution, inheritance chain resolution, packing execution | Backend admin uploads with deduplication | **No cleanup** - Permanent storage | Reference resolution in recipes, inheritance chain resolution | Handles inheritance chains | +| **objects** | Stores object definitions (molecules, organelles) | `name`, `id`, `dedup_hash`, `type`, `color`, `packing_mode`, `place_method`, `radius`, `inherit`, `gradient`, `representations`, `jitter_attempts` | Recipe reference resolution, inheritance chain resolution, packing execution | Backend admin uploads with deduplication | **No cleanup** - Permanent storage | Reference resolution in recipes, inheritance chain resolution | | | **gradients** | Stores gradient definitions for spatial distributions | `name`, `id`, `dedup_hash`, `description`, `mode`, `mode_settings`, `weight_mode`, `pick_mode`, `reversed`, `invert` | Recipe/object reference resolution, editable field options | Backend admin uploads with deduplication | **No cleanup** - Permanent storage | Reference resolution, gradient options for editable fields | Can be combined (multiple per object) | -| **composition** | Stores composition definitions (what objects go where) | `name`, `id`, `dedup_hash`, `count`, `molarity`, `object`, `inherit`, `regions` (interior, surface, leaflets) | Recipe reference resolution, region-based placement | Backend admin uploads with topological sort | **No cleanup** - Permanent storage | Reference resolution, region-based object placement | Handles nested composition dependencies | -| **job_status** | Tracks AWS packing job status | `status`, `error_message`, `outputs_directory`, `result_path`, `timestamp` | Frontend polling (every 500ms), progress monitoring, result retrieval | Backend updates during job execution (RUNNING → DONE/FAILED) | **No automated cleanup** - Manual only | Job monitoring, status polling, result retrieval | Not in backend default collection list | -| **example_packings** | Maps example recipes to configs and defines UI metadata | `name` (display name), `recipe` (recipe ID), `config` (config ID), `editable_fields` (array of field IDs), `result_path` | App startup, recipe dropdown population | Manual admin setup (for frontend only) | **No cleanup** - Permanent storage | App initialization, recipe dropdown population, UI configuration | **Frontend only** - Not used by backend | -| **editable_fields** | Defines which recipe fields users can edit in UI | `id`, `name`, `data_type`, `input_type`, `description`, `path`, `min`, `max`, `options`, `gradient_options`, `conversion_factor`, `unit` | App initialization, form generation, input validation | Manual admin setup (for frontend only) | **No cleanup** - Permanent storage | Dynamic form generation, input validation, gradient options | **Frontend only** - Not used by backend | +| **composition** | Stores composition definitions (what objects go where) | `name`, `id`, `dedup_hash`, `count`, `molarity`, `object`, `inherit`, `regions` (interior, surface, leaflets) | Recipe reference resolution, region-based placement | Backend admin uploads with topological sort | **No cleanup** - Permanent storage | Reference resolution, region-based object placement | | +| **job_status** | Tracks AWS packing job status | `status`, `error_message`, `outputs_directory`, `result_path`, `timestamp` | Frontend polling (every 500ms), progress monitoring, result retrieval | Backend updates during job execution (RUNNING → DONE/FAILED) | **24 hours** - cleaned if older than 24 hours | Job monitoring, status polling, result retrieval | | +| **example_packings** | Maps example recipes to configs and defines UI metadata | `name` (display name), `recipe` (recipe ID), `config` (config ID), `editable_fields` (array of field IDs), `result_path` | App startup, recipe dropdown population | The collection to receive recipes that users want to pack in Studio | **No cleanup** - Permanent storage | App initialization, recipe dropdown population, UI configuration | | +| **editable_fields** | Defines which recipe fields users can edit in UI | `id`, `name`, `data_type`, `input_type`, `description`, `path`, `min`, `max`, `options`, `gradient_options`, `conversion_factor`, `unit` | App initialization, form generation, input validation | Defined by users through upload script | **No cleanup** - Permanent storage | Dynamic form generation, input validation, gradient options | **Details for upload** - [documentation](https://github.com/mesoscope/cellpack/blob/main/docs/STUDIO_SITE.md#cellpack-studio-site) | | **results** | Stores Simularium result metadata for auto-opening (backend only) | `batch_job_id`, `timestamp`, `url` (Simularium result S3 URL), `user` | Backend cleanup validation (S3 URL check) | Successful packing completion with Simularium output | **180 days** - Cleaned if S3 URL inaccessible | Result archiving, user history tracking, result URL storage | Cleanup validates S3 existence | From a89d202b53d4ccc35a742f9672443548923cd640 Mon Sep 17 00:00:00 2001 From: Ruge Li Date: Fri, 24 Apr 2026 10:32:23 -0700 Subject: [PATCH 3/9] doc update: remove recipes_edited --- CONTRIBUTING.md | 7 +++---- FIREBASE_SCHEMA.md | 5 ++--- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e059ef7f..ceb87be6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -19,7 +19,7 @@ you'd like to see documented. ## Firebase Overview -cellPACK Studio reads recipe data from Firebase and writes user edits back. The backend handles uploading recipes and running packing jobs. +cellPACK Studio reads recipe data from Firebase to render the editor and polls job status during a packing. When a user submits an edited recipe, the client sends the modified recipe as JSON in the packing-request body and the backend processes it directly. ### Collections Used by cellPACK Studio @@ -31,10 +31,9 @@ cellPACK Studio reads recipe data from Firebase and writes user edits back. The | `objects` | Object definitions (molecules, organelles) | Read | | `gradients` | Gradient definitions for spatial distributions | Read | | `composition` | Composition definitions | Read | -| `recipes_edited` | User-modified recipes | Write | -| `job_status` | Packing job progress | Poll (read) | +| `job_status` | Packing job progress | Poll (read), refresh `timestamp`, cleanup | -> **Note:** Collections like `configs` and `results` are managed by the backend. For the complete database schema, see [FIREBASE_SCHEMA.md](FIREBASE_SCHEMA.md). +> **Note:** Collections like `configs` and `results` are managed by the backend. Scheduled cleanup of `job_status` is run by this repo via `.github/workflows/cleanup.yml` (see `scripts/cleanup.ts`). For the complete database schema, see [FIREBASE_SCHEMA.md](FIREBASE_SCHEMA.md). ### Getting Access diff --git a/FIREBASE_SCHEMA.md b/FIREBASE_SCHEMA.md index ec703442..845055a1 100644 --- a/FIREBASE_SCHEMA.md +++ b/FIREBASE_SCHEMA.md @@ -7,13 +7,12 @@ This document describes the Firestore database schema used by cellPACK. Collecti | Collection Name | Purpose | Key Fields | Read | Write | Cleanup Policy | Related Process | Notes | | -------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------- | ---------------------------------------------------------------- | ----------------------------------------------- | | **recipes** | Stores the full-structured recipes | `name`, `id`, `dedup_hash`, `description`, `version`, `format_version`, `bounding_box`, `composition`, `objects`, `gradients`, `optional_gradients` | Recipe loading, UI display, user edit comparison, packing execution | Backend uploads via admin CLI tool | **No cleanup** - Permanent storage | Recipe loading, reference resolution, comparison with user edits | `optional_gradients` manually added | -| **recipes_edited** | Stores user-modified recipes temporarily | Same as `recipes` + `recipe_path`, `timestamp` | Backend retrieval for packing | User submits modified recipe (differs from default) | **24 hours** - cleaned if older than 24 hours | User recipe submission, packing job submission | Unnested format, no reference resolution needed | | **configs** | Stores packing configuration settings | Referenced by path (e.g., `firebase:configs/{id}`) | Building packing job request, backend reads during packing | Backend admin uploads via CLI tool | **No cleanup** - Permanent storage | Packing job configuration, submitted to Docker server | Referenced but not directly queried by frontend | | **objects** | Stores object definitions (molecules, organelles) | `name`, `id`, `dedup_hash`, `type`, `color`, `packing_mode`, `place_method`, `radius`, `inherit`, `gradient`, `representations`, `jitter_attempts` | Recipe reference resolution, inheritance chain resolution, packing execution | Backend admin uploads with deduplication | **No cleanup** - Permanent storage | Reference resolution in recipes, inheritance chain resolution | | | **gradients** | Stores gradient definitions for spatial distributions | `name`, `id`, `dedup_hash`, `description`, `mode`, `mode_settings`, `weight_mode`, `pick_mode`, `reversed`, `invert` | Recipe/object reference resolution, editable field options | Backend admin uploads with deduplication | **No cleanup** - Permanent storage | Reference resolution, gradient options for editable fields | Can be combined (multiple per object) | | **composition** | Stores composition definitions (what objects go where) | `name`, `id`, `dedup_hash`, `count`, `molarity`, `object`, `inherit`, `regions` (interior, surface, leaflets) | Recipe reference resolution, region-based placement | Backend admin uploads with topological sort | **No cleanup** - Permanent storage | Reference resolution, region-based object placement | | -| **job_status** | Tracks AWS packing job status | `status`, `error_message`, `outputs_directory`, `result_path`, `timestamp` | Frontend polling (every 500ms), progress monitoring, result retrieval | Backend updates during job execution (RUNNING → DONE/FAILED) | **24 hours** - cleaned if older than 24 hours | Job monitoring, status polling, result retrieval | | -| **example_packings** | Maps example recipes to configs and defines UI metadata | `name` (display name), `recipe` (recipe ID), `config` (config ID), `editable_fields` (array of field IDs), `result_path` | App startup, recipe dropdown population | The collection to receive recipes that users want to pack in Studio | **No cleanup** - Permanent storage | App initialization, recipe dropdown population, UI configuration | | +| **job_status** | Tracks AWS packing job status | `status`, `error_message`, `outputs_directory`, `result_path`, `timestamp` | Frontend polling (every 500ms), progress monitoring, result retrieval | Backend updates during job execution (RUNNING → DONE/FAILED); frontend refreshes `timestamp` after the final read | **30 days** - cleaned if older than 30 days | Job monitoring, status polling, result retrieval | | +| **example_packings** | Maps example recipes to configs and defines UI metadata | `name` (display name), `recipe` (recipe ID), `config` (config ID), `editable_fields` (array of field IDs), `result_path`, `outputs_directory` | App startup, recipe dropdown population | The collection to receive recipes that users want to pack in Studio | **No cleanup** - Permanent storage | App initialization, recipe dropdown population, UI configuration | | | **editable_fields** | Defines which recipe fields users can edit in UI | `id`, `name`, `data_type`, `input_type`, `description`, `path`, `min`, `max`, `options`, `gradient_options`, `conversion_factor`, `unit` | App initialization, form generation, input validation | Defined by users through upload script | **No cleanup** - Permanent storage | Dynamic form generation, input validation, gradient options | **Details for upload** - [documentation](https://github.com/mesoscope/cellpack/blob/main/docs/STUDIO_SITE.md#cellpack-studio-site) | | **results** | Stores Simularium result metadata for auto-opening (backend only) | `batch_job_id`, `timestamp`, `url` (Simularium result S3 URL), `user` | Backend cleanup validation (S3 URL check) | Successful packing completion with Simularium output | **180 days** - Cleaned if S3 URL inaccessible | Result archiving, user history tracking, result URL storage | Cleanup validates S3 existence | From bee0a1b1f2b8e4316fbccdaa1731e2120edf1791 Mon Sep 17 00:00:00 2001 From: Ruge Li <91452427+rugeli@users.noreply.github.com> Date: Tue, 28 Apr 2026 11:19:04 -0700 Subject: [PATCH 4/9] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- FIREBASE_SCHEMA.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/FIREBASE_SCHEMA.md b/FIREBASE_SCHEMA.md index 845055a1..c71ab11b 100644 --- a/FIREBASE_SCHEMA.md +++ b/FIREBASE_SCHEMA.md @@ -1,6 +1,6 @@ # Firebase Schema Documentation -This document describes the Firestore database schema used by cellPACK. Collections are accessed by the client (cellPACK Studio), the core package(backend), or both. +This document describes the Firestore database schema used by cellPACK. Collections are accessed by the client (cellPACK Studio), the core package (backend), or both. ## Collections From 61223e3334f5a6d3a9ebf7947106556949bd7b6f Mon Sep 17 00:00:00 2001 From: Ruge Li Date: Tue, 28 Apr 2026 13:47:33 -0700 Subject: [PATCH 5/9] clarify wording --- FIREBASE_SCHEMA.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/FIREBASE_SCHEMA.md b/FIREBASE_SCHEMA.md index c71ab11b..ed0d6a1a 100644 --- a/FIREBASE_SCHEMA.md +++ b/FIREBASE_SCHEMA.md @@ -12,7 +12,7 @@ This document describes the Firestore database schema used by cellPACK. Collecti | **gradients** | Stores gradient definitions for spatial distributions | `name`, `id`, `dedup_hash`, `description`, `mode`, `mode_settings`, `weight_mode`, `pick_mode`, `reversed`, `invert` | Recipe/object reference resolution, editable field options | Backend admin uploads with deduplication | **No cleanup** - Permanent storage | Reference resolution, gradient options for editable fields | Can be combined (multiple per object) | | **composition** | Stores composition definitions (what objects go where) | `name`, `id`, `dedup_hash`, `count`, `molarity`, `object`, `inherit`, `regions` (interior, surface, leaflets) | Recipe reference resolution, region-based placement | Backend admin uploads with topological sort | **No cleanup** - Permanent storage | Reference resolution, region-based object placement | | | **job_status** | Tracks AWS packing job status | `status`, `error_message`, `outputs_directory`, `result_path`, `timestamp` | Frontend polling (every 500ms), progress monitoring, result retrieval | Backend updates during job execution (RUNNING → DONE/FAILED); frontend refreshes `timestamp` after the final read | **30 days** - cleaned if older than 30 days | Job monitoring, status polling, result retrieval | | -| **example_packings** | Maps example recipes to configs and defines UI metadata | `name` (display name), `recipe` (recipe ID), `config` (config ID), `editable_fields` (array of field IDs), `result_path`, `outputs_directory` | App startup, recipe dropdown population | The collection to receive recipes that users want to pack in Studio | **No cleanup** - Permanent storage | App initialization, recipe dropdown population, UI configuration | | +| **example_packings** | Maps example recipes to configs and defines UI metadata | `name` (display name), `recipe` (recipe ID), `config` (config ID), `editable_fields` (array of field IDs), `result_path`, `outputs_directory` | App startup, recipe dropdown population | Populated by recipe authors via the Studio upload script | **No cleanup** - Permanent storage | App initialization, recipe dropdown population, UI configuration | | | **editable_fields** | Defines which recipe fields users can edit in UI | `id`, `name`, `data_type`, `input_type`, `description`, `path`, `min`, `max`, `options`, `gradient_options`, `conversion_factor`, `unit` | App initialization, form generation, input validation | Defined by users through upload script | **No cleanup** - Permanent storage | Dynamic form generation, input validation, gradient options | **Details for upload** - [documentation](https://github.com/mesoscope/cellpack/blob/main/docs/STUDIO_SITE.md#cellpack-studio-site) | | **results** | Stores Simularium result metadata for auto-opening (backend only) | `batch_job_id`, `timestamp`, `url` (Simularium result S3 URL), `user` | Backend cleanup validation (S3 URL check) | Successful packing completion with Simularium output | **180 days** - Cleaned if S3 URL inaccessible | Result archiving, user history tracking, result URL storage | Cleanup validates S3 existence | From ae5b34f77b8c961d792c4c8d674d620696b50c44 Mon Sep 17 00:00:00 2001 From: Ruge Li Date: Tue, 28 Apr 2026 13:55:46 -0700 Subject: [PATCH 6/9] remove "results" collection --- CONTRIBUTING.md | 2 +- FIREBASE_SCHEMA.md | 1 - README.md | 4 ++-- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ceb87be6..65b8b938 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -33,7 +33,7 @@ cellPACK Studio reads recipe data from Firebase to render the editor and polls j | `composition` | Composition definitions | Read | | `job_status` | Packing job progress | Poll (read), refresh `timestamp`, cleanup | -> **Note:** Collections like `configs` and `results` are managed by the backend. Scheduled cleanup of `job_status` is run by this repo via `.github/workflows/cleanup.yml` (see `scripts/cleanup.ts`). For the complete database schema, see [FIREBASE_SCHEMA.md](FIREBASE_SCHEMA.md). +> **Note:** The `configs` collection is managed by the backend. Scheduled cleanup of `job_status` is run by this repo via `.github/workflows/cleanup.yml` (see `scripts/cleanup.ts`). For the complete database schema, see [FIREBASE_SCHEMA.md](FIREBASE_SCHEMA.md). ### Getting Access diff --git a/FIREBASE_SCHEMA.md b/FIREBASE_SCHEMA.md index ed0d6a1a..18d5832d 100644 --- a/FIREBASE_SCHEMA.md +++ b/FIREBASE_SCHEMA.md @@ -14,5 +14,4 @@ This document describes the Firestore database schema used by cellPACK. Collecti | **job_status** | Tracks AWS packing job status | `status`, `error_message`, `outputs_directory`, `result_path`, `timestamp` | Frontend polling (every 500ms), progress monitoring, result retrieval | Backend updates during job execution (RUNNING → DONE/FAILED); frontend refreshes `timestamp` after the final read | **30 days** - cleaned if older than 30 days | Job monitoring, status polling, result retrieval | | | **example_packings** | Maps example recipes to configs and defines UI metadata | `name` (display name), `recipe` (recipe ID), `config` (config ID), `editable_fields` (array of field IDs), `result_path`, `outputs_directory` | App startup, recipe dropdown population | Populated by recipe authors via the Studio upload script | **No cleanup** - Permanent storage | App initialization, recipe dropdown population, UI configuration | | | **editable_fields** | Defines which recipe fields users can edit in UI | `id`, `name`, `data_type`, `input_type`, `description`, `path`, `min`, `max`, `options`, `gradient_options`, `conversion_factor`, `unit` | App initialization, form generation, input validation | Defined by users through upload script | **No cleanup** - Permanent storage | Dynamic form generation, input validation, gradient options | **Details for upload** - [documentation](https://github.com/mesoscope/cellpack/blob/main/docs/STUDIO_SITE.md#cellpack-studio-site) | -| **results** | Stores Simularium result metadata for auto-opening (backend only) | `batch_job_id`, `timestamp`, `url` (Simularium result S3 URL), `user` | Backend cleanup validation (S3 URL check) | Successful packing completion with Simularium output | **180 days** - Cleaned if S3 URL inaccessible | Result archiving, user history tracking, result URL storage | Cleanup validates S3 existence | diff --git a/README.md b/README.md index 61a4de0d..d25bd389 100644 --- a/README.md +++ b/README.md @@ -22,13 +22,13 @@ This client interacts with the cellPACK server, which consists of a variety of b * POST /submit-packing?recipe={myrecipe}&config={myconfig} * GET /logs?logStreamName={name} * GET /packing-status?jobId={id} -* **Batch**: A call to POST /submit-packing launches a new batch job to run. A call to GET /packing-status will return the status of the specified batch job. Once the job is completed, the path to the results file(s) will be added to the results table of the cellPACK Firebase database. +* **Batch**: A call to POST /submit-packing launches a new batch job to run. A call to GET /packing-status will return the status of the specified batch job. Once the job is completed, the path to the results file(s) is written to the job's `job_status` entry in the cellPACK Firebase database. * **S3**: Result files from the AWS Batch job are written to the `cellpack-demo` S3 bucket. * **ECR**: Docker image built from the [cellPACK github repo](https://github.com/mesoscope/cellpack) is published to the `cellpack-private` ECR repository. That image defines the container specificationsin which the batch job will run. * **CloudWatch**: Logs from each AWS Batch job are written to CloudWatch. These logs can be accessed via the GET /logs endpoint. ### cellPACK Database -* **Firebase Firestore**: The cellPACK database is hosted in Firebase Firestore. This database stores all recipes, objects, gradients, compositions, packing configurations, job statuses, and results metadata. See [CONTRIBUTING.md](CONTRIBUTING.md#firebase-overview) for Firebase overview and [FIREBASE_SCHEMA.md](FIREBASE_SCHEMA.md) for the complete database schema. +* **Firebase Firestore**: The cellPACK database is hosted in Firebase Firestore. This database stores all recipes, objects, gradients, compositions, packing configurations, and job statuses. See [CONTRIBUTING.md](CONTRIBUTING.md#firebase-overview) for Firebase overview and [FIREBASE_SCHEMA.md](FIREBASE_SCHEMA.md) for the complete database schema. #### Resources * [Server Architecture Overview Diagram](https://docs.google.com/presentation/d/1eG2XCxgYNaoDIYI-M6Tzef17bGuFirZZhmfZlBFTaXc/edit#slide=id.g26c8fd413da_0_34) From 57b27b8cb37892a9a7550facb4a7d4d3060b41fc Mon Sep 17 00:00:00 2001 From: Ruge Li Date: Tue, 28 Apr 2026 14:16:32 -0700 Subject: [PATCH 7/9] manage links --- CONTRIBUTING.md | 2 +- FIREBASE_SCHEMA.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 65b8b938..b7bd0008 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -38,7 +38,7 @@ cellPACK Studio reads recipe data from Firebase to render the editor and polls j ### Getting Access - **Development:** Create your own Firebase project in test mode. See [Firebase Firestore tutorial](https://firebase.google.com/docs/firestore). -- **Staging:** Contact the code owner for credentials, then configure your `.env` file (see README). +- **Staging:** Contact the code owner for credentials, then configure your `.env` file. ## How to Contribute diff --git a/FIREBASE_SCHEMA.md b/FIREBASE_SCHEMA.md index 18d5832d..343345ad 100644 --- a/FIREBASE_SCHEMA.md +++ b/FIREBASE_SCHEMA.md @@ -12,6 +12,6 @@ This document describes the Firestore database schema used by cellPACK. Collecti | **gradients** | Stores gradient definitions for spatial distributions | `name`, `id`, `dedup_hash`, `description`, `mode`, `mode_settings`, `weight_mode`, `pick_mode`, `reversed`, `invert` | Recipe/object reference resolution, editable field options | Backend admin uploads with deduplication | **No cleanup** - Permanent storage | Reference resolution, gradient options for editable fields | Can be combined (multiple per object) | | **composition** | Stores composition definitions (what objects go where) | `name`, `id`, `dedup_hash`, `count`, `molarity`, `object`, `inherit`, `regions` (interior, surface, leaflets) | Recipe reference resolution, region-based placement | Backend admin uploads with topological sort | **No cleanup** - Permanent storage | Reference resolution, region-based object placement | | | **job_status** | Tracks AWS packing job status | `status`, `error_message`, `outputs_directory`, `result_path`, `timestamp` | Frontend polling (every 500ms), progress monitoring, result retrieval | Backend updates during job execution (RUNNING → DONE/FAILED); frontend refreshes `timestamp` after the final read | **30 days** - cleaned if older than 30 days | Job monitoring, status polling, result retrieval | | -| **example_packings** | Maps example recipes to configs and defines UI metadata | `name` (display name), `recipe` (recipe ID), `config` (config ID), `editable_fields` (array of field IDs), `result_path`, `outputs_directory` | App startup, recipe dropdown population | Populated by recipe authors via the Studio upload script | **No cleanup** - Permanent storage | App initialization, recipe dropdown population, UI configuration | | +| **example_packings** | Maps example recipes to configs and defines UI metadata | `name` (display name), `recipe` (recipe ID), `config` (config ID), `editable_fields` (array of field IDs), `result_path`, `outputs_directory` | App startup, recipe dropdown population | Populated by recipe authors via the Studio upload script | **No cleanup** - Permanent storage | App initialization, recipe dropdown population, UI configuration | **Details for upload** - [documentation](https://github.com/mesoscope/cellpack/blob/main/docs/STUDIO_SITE.md#cellpack-studio-site) | | **editable_fields** | Defines which recipe fields users can edit in UI | `id`, `name`, `data_type`, `input_type`, `description`, `path`, `min`, `max`, `options`, `gradient_options`, `conversion_factor`, `unit` | App initialization, form generation, input validation | Defined by users through upload script | **No cleanup** - Permanent storage | Dynamic form generation, input validation, gradient options | **Details for upload** - [documentation](https://github.com/mesoscope/cellpack/blob/main/docs/STUDIO_SITE.md#cellpack-studio-site) | From d74aee54750abf2dbbb79ac2af72c2983220b3f4 Mon Sep 17 00:00:00 2001 From: Ruge Li Date: Wed, 29 Apr 2026 16:33:42 -0700 Subject: [PATCH 8/9] remove batch info --- README.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index d25bd389..74b46894 100644 --- a/README.md +++ b/README.md @@ -19,19 +19,15 @@ Front end website to interact with the cellPACK services running in AWS. This we ### cellPACK Server This client interacts with the cellPACK server, which consists of a variety of backend services hosted in AWS to run [cellPACK packings](https://github.com/mesoscope/cellpack). These AWS services include: * **API Gateway**: cellPACK REST API providing this client with access to needed AWS resources for running and receiving data from cellPACK jobs. Includes the following endpoints: - * POST /submit-packing?recipe={myrecipe}&config={myconfig} - * GET /logs?logStreamName={name} - * GET /packing-status?jobId={id} -* **Batch**: A call to POST /submit-packing launches a new batch job to run. A call to GET /packing-status will return the status of the specified batch job. Once the job is completed, the path to the results file(s) is written to the job's `job_status` entry in the cellPACK Firebase database. -* **S3**: Result files from the AWS Batch job are written to the `cellpack-demo` S3 bucket. -* **ECR**: Docker image built from the [cellPACK github repo](https://github.com/mesoscope/cellpack) is published to the `cellpack-private` ECR repository. That image defines the container specificationsin which the batch job will run. -* **CloudWatch**: Logs from each AWS Batch job are written to CloudWatch. These logs can be accessed via the GET /logs endpoint. + * POST /start-packing?recipe={myrecipe}&config={myconfig} +* **ECS**: A call to POST /start-packing launches a new AWS packing job to run. Once the job is completed, the path to the results file(s) is written to the job's `job_status` entry in the cellPACK Firebase database. +* **S3**: Result files from the AWS packing job are written to the `cellpack-demo` S3 bucket. +* **ECR**: Docker image built from the [cellPACK github repo](https://github.com/mesoscope/cellpack) is published to the `cellpack-private` ECR repository. That image defines the container specifications in which the AWS packing job will run. ### cellPACK Database * **Firebase Firestore**: The cellPACK database is hosted in Firebase Firestore. This database stores all recipes, objects, gradients, compositions, packing configurations, and job statuses. See [CONTRIBUTING.md](CONTRIBUTING.md#firebase-overview) for Firebase overview and [FIREBASE_SCHEMA.md](FIREBASE_SCHEMA.md) for the complete database schema. #### Resources * [Server Architecture Overview Diagram](https://docs.google.com/presentation/d/1eG2XCxgYNaoDIYI-M6Tzef17bGuFirZZhmfZlBFTaXc/edit#slide=id.g26c8fd413da_0_34) -* [AWS Batch Dashboard](https://us-west-2.console.aws.amazon.com/batch/home?region=us-west-2#) * [Staging Firebase Database](https://console.firebase.google.com/u/0/project/cell-pack-database/firestore/databases/-default-/data/~2Fcomposition~2F9XjxZ0ApsNQqCXUbtVTc) * [Project Notes](https://docs.google.com/document/d/1jqIvf8DzjWgzbG-NMMJ8pdQbi2qQ7wrzLdLqyR7F0i0/edit?tab=t.0#heading=h.yg9wht4r88xr) From b2a7b290f4484bbb7153cb60b88fb84ca0b322fa Mon Sep 17 00:00:00 2001 From: Ruge Li Date: Wed, 29 Apr 2026 16:42:37 -0700 Subject: [PATCH 9/9] mark config as an optional part --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 74b46894..35f87611 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Front end website to interact with the cellPACK services running in AWS. This we ### cellPACK Server This client interacts with the cellPACK server, which consists of a variety of backend services hosted in AWS to run [cellPACK packings](https://github.com/mesoscope/cellpack). These AWS services include: * **API Gateway**: cellPACK REST API providing this client with access to needed AWS resources for running and receiving data from cellPACK jobs. Includes the following endpoints: - * POST /start-packing?recipe={myrecipe}&config={myconfig} + * POST /start-packing?recipe={myrecipe}[&config={myconfig}] * **ECS**: A call to POST /start-packing launches a new AWS packing job to run. Once the job is completed, the path to the results file(s) is written to the job's `job_status` entry in the cellPACK Firebase database. * **S3**: Result files from the AWS packing job are written to the `cellpack-demo` S3 bucket. * **ECR**: Docker image built from the [cellPACK github repo](https://github.com/mesoscope/cellpack) is published to the `cellpack-private` ECR repository. That image defines the container specifications in which the AWS packing job will run.