Datavis analysis - #17
Open
A-Jolly-Holly wants to merge 36 commits into
Open
Conversation
…pids - Gene expression source resolves per group: layers/gexp (csc or dense, the zarrsmith add-expr shape) -> dense X -> CSC X. CSR-only stores get a clear error pointing at zarrsmith add-expr. - CSC columns read as two range requests (data/indices sliced by a cached indptr) scattered into a dense per-cell vector; BigInt64 indices handled. - RAPIDS_DIR env (optional): the app displays everything from DATA_DIR; selection downloads and submissions record RAPIDS_DIR as their store (barcodes remain the join key). Unset, one store serves both. Passed through docker compose; /api/config exposes both. Verified end-to-end against generated fixture stores: gexp-layer, CSC-X, and dense-X paths all return exact expected values through the backend proxy; frontend builds clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found while benchmarking the 3M-cell GCS stores: zarr.open re-fetched data/indices zarr.json on every gene click (~2 extra proxied GCS round trips), cutting CSC gene reads from ~205ms to ~140ms median. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… + reverted - Backend serves RAPIDS_DIR at /api/rapids-data (same proxy semantics); with two stores the app reads obsm/obs/groups.json from the rapids store and only gene expression from DATA_DIR. Verified with sentinel coords: rapids-data serves coords/labels, data serves gexp values. - Config fetch is origin-absolute; nginx /api proxy_buffering off. - Streaming the GCS relay was implemented two ways (BlobReader chunks, download_to_file queue relay) and benchmarked against the 3M-cell stores: no improvement — chunks compress to ~4 KB, so the dense gene cost is browser-side decode/assembly over 3M rows, not transfer. Reverted to the buffered one-liner; finding recorded in the handler. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Raw X sources are right-skewed: linear min-max let a few high-count cells own the ramp and everything else sat near the zero color. Display value is now log1p(v) clipped at the 99th percentile of the non-zeros (scanpy-style); gradient labels stay in raw counts with the top label = the clip. layers/gexp sources are already normalized and are rendered unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This reverts commit a6899d5.
Integer dtype on the expression source = un-normalized counts; the GenePicker shows an amber note under the gradient. No per-value work — the dtype comes from the already-cached array metadata. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Backend job queue (one A100, serial): scp selection + parameterized rerun script to the GCE box, run under the rapids pixi env via gcloud compute ssh, stream stage: lines into the job record, rsync the view store into the rapids GCS store's umap_views/<slug>, and register it in groups.json. Falls back to the sleep simulator when GPU_INSTANCE is unset (tests unchanged). Config: GPU_INSTANCE, GPU_ZONE, GPU_DATA, GPU_PIXI_DIR. - rerun_umap_on_selection.py: env-var overrides (RERUN_DATA/SELECTION/ OUT/GPUS), stage prints for monitoring, GPU-count-aware rechunk; manual defaults unchanged. - Frontend: button renamed to "Generate new UMAP with subset" with a view-name field; submit payload carries barcodes again (the GPU side selects by barcode, view-safe); runs panel shows queued/running with a live elapsed timer + current pipeline stage, and on completion the app reloads groups.json and jumps to the new view. Verified end-to-end: 2000-cell selection -> queued -> pipeline stages -> view in GCS -> groups.json entry -> coords readable through /api/rapids-data, 96s total on the A100. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- rerun script: selections under RERUN_EAGER_MAX (150k cells) skip the
dask-cuda cluster entirely — eager sparse_dataset row load + in-core
cupy pipeline on one GPU; larger selections keep the dask path.
- backend runner: 5 remote invocations -> 2 (one scp ships selection +
script; one ssh chains pipeline -> GCS rsync -> cleanup, streaming
stage lines throughout).
- DELETE /api/views/{id}: removes the view's objects from the rapids
GCS store and its groups.json entry; small confirm-guarded x button
in the View picker when a view is active.
Re-measured end-to-end at 2000 cells: ~50s submit-to-registered-view.
Delete verified against real views (92 objects each, groups restored).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- gpu_runner.py (in-repo): warm process on the box with RAPIDS imported and obs/var cached; watches /tmp/datavis_jobs/ for selection JSONs, runs the eager single-GPU pipeline, writes the view directly to gs:// (gcloud rsync fallback), streams stage lines + done/failed/fallback. Selections over RUNNER_EAGER_MAX hand back to the dask-cuda cold path. - backend: one ssh per job — pipes the selection JSON via stdin into the watched dir, auto-(re)starts the runner on stale heartbeat, tails the job log for live stages; scripts shipped from the repo checkout once per backend process. Cold fallback preserved for large jobs. - gpu_bootstrap.sh: one command on any fresh GPU box — pixi install, repo clone, RAPIDS env build, runner start. Nothing instance-specific beyond the backend's GPU_INSTANCE/GPU_ZONE/GPU_DATA env. Measured: warm job 20s submit-to-registered-view (2000 cells); first job after box cold-start ~32s including runner warm-up. Note: direct gs:// writes work with the instance SA today; gs:// *input* (GPU_DATA) still needs objects.list on the bucket. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One instance is the reality; GPU_INSTANCE now defaults to it (empty string selects the simulator), GPU_DATA defaults to the box's /mnt store and is passable through docker compose. README documents the GPU-run config and the runner-restart-after-store-change note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The compose one-liner is now the whole startup, GPU jobs included: backend image installs google-cloud-cli + openssh-client; entrypoint copies the read-only-mounted host gcloud config and compute-engine ssh key into place; gpu_runner.py / rerun script are volume-mounted from the repo so the container ships the checked-in copies. GPU_INSTANCE defaults preserved in compose (empty no longer silently selects the simulator). Verified: e2e job submitted through the container, 18s submit-to-view on the warm runner. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Opening :8000 in a browser 404'd (API-only port, docs removed), which read as a failed deploy; the root now answers with a pointer to :3000. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Health checks stay at /api/health; /docs is the human console for poking submit/jobs/views. Root signpost links both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- delete_view unregisters the groups.json entry before deleting objects: an interrupted delete now leaves unlisted orphans instead of a listed-but-broken view. - The fatal load-error screen gained a Retry button (a backend restart mid-request previously dead-ended the app on a transient 502). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…very - delete_view: GCS batch API (100/round-trip, was 1 HTTPS DELETE per object, ~10-30s for a view); idempotent orphan sweep when unlisted; groups.json writes serialized under a lock vs the GPU worker's register; size-cache purge - groups.json served Cache-Control: no-store (both GCS and local paths) and fetched no-store, so a cached listing can't resurrect deleted views - delete UX: relocate to full store immediately, pointer-blocking scrim while the delete runs, failures surface in a dismissible banner (submit too) - selecting a stale view now refreshes the listing and falls back to the full store instead of the dead-end error screen - coords/categorical/gene-name effects get stale-response guards (a slow previous-group response could corrupt selection artifacts) - gene reads in views resolve against the coord store (views live there in two-store deployments); frontend caches evict on rejection instead of poisoning the session; BigInt-safe dense gene reads; /api/config failures no longer cached; queued/running jobs exempt from JOBS eviction Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…inished views Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A failed AIFI-level load (e.g. a backend restart mid-page-load) previously left the Legend on 'Loading…' forever with only a console.error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…odules - server/ split: config (env), storage (zarr proxy + store JSON helpers), views (groups.json lifecycle), gpu (job queue + dispatch), main (routes only) - one DATA_DIR store for everything: proxied reads, jobs/ queue + status, umap_views/; RAPIDS_DIR / GPU_DATA / /api/rapids-data removed - GPU runs are cold per job, cluster-style: backend writes jobs/submitted/<id>.json to the store, ships gpu_job.sh + rerun_umap_on_selection.py fresh, one ssh runs them; the script does env setup, the pipeline against gs:// (gcsfs + ADC), view upload, and streams every stage to jobs/status/<id>.json which the backend polls for the UI - warm gpu_runner.py retired (deleted; process + files removed from the box) - rerun script: zarr read concurrency raised for cold GCS loads - verified e2e on the 3M store: 50k-cell selection, submit -> registered view in 48s cold; view delete 97 objects in 3.4s Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dark; adaptive point size Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ubsets Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…threads, zarr config on client+workers) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…I_DIR in compose Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he rerun script Drops the RERUN_* env forwarding — rapids arguments are defined where the pipeline runs, not sent over per job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rom .env only Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- submit 400s (with detail surfaced in the UI banner) when GPU_INSTANCE/ZONE/ PIXI_DIR are unset instead of silently simulating; simulator removed - gcloud -q on ssh/scp: first run on a fresh machine auto-generates the keypair - compose mounts ~/.ssh as a dir (key files may not exist yet) - gpu_job.sh bootstraps pixi for a fresh user account on the box - README: real IAM requirements (objectAdmin + compute ssh) and the one-time service-account grant that makes on-box store access user-independent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
A-Jolly-Holly
force-pushed
the
datavis_analysis
branch
from
September 2, 2026 16:26
6246ae8 to
183db11
Compare
…ding Strict per-segment whitelist (zarr member names are plain: alnum . _ -, no leading dot) before any filesystem use, and resolve/stat wrapped so malformed paths 404 instead of 500. Traversal was already blocked by the ..-segment check plus resolve+is_relative_to containment; this makes hostile input unrepresentable before path construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Working on realtime visualization app. Currently needs 1 GCS zarr store, and a GPU instance. Set in .env for config.