From b1882b8af2d92657d09ae61686eec442ddf7e27e Mon Sep 17 00:00:00 2001 From: Glen Beane <356266+gbeane@users.noreply.github.com> Date: Tue, 28 Jul 2026 15:38:11 -0400 Subject: [PATCH 1/5] Add JABS Hub-backed projects client integration plan --- .../plans/jabs-hub-backed-projects-plan.md | 608 ++++++++++++++++++ 1 file changed, 608 insertions(+) create mode 100644 docs/development/plans/jabs-hub-backed-projects-plan.md diff --git a/docs/development/plans/jabs-hub-backed-projects-plan.md b/docs/development/plans/jabs-hub-backed-projects-plan.md new file mode 100644 index 00000000..6e117b00 --- /dev/null +++ b/docs/development/plans/jabs-hub-backed-projects-plan.md @@ -0,0 +1,608 @@ +# Plan: Hub-Backed JABS Projects — Client Integration (jabs-behavior-classifier) + +**Status:** Draft / planning +**Author:** Glen Beane (with Claude Code) +**Repository:** `jabs-behavior-classifier` (this repo — the PySide6 desktop GUI / Python client). +**Companion doc:** the Hub-side (Go) design lives in the `jabs-hub` repo at +`docs/specs/0006-jabs-project-backend-design.md`. The REST API, the PostgreSQL data model, and +the annotation-document schema (Appendix A.1) are the **shared contract** — keep them in sync. +**Scope:** The client half of Hub-backed JABS projects under a **cloud-native** model: the desktop +GUI becomes a caching client that pulls videos, pose files, annotations, and project metadata from +JABS Hub / Google Cloud Storage into a local cache, works against the cache (frame-accurate +labeling, feature extraction, training, offline), and pushes changes back. + +--- + +## 1. Overview — JABS Hub-backed projects + +A **Hub-backed JABS project is a cloud project**: its videos, pose files, behavior annotations, +and project metadata all live in **JABS Hub** (PostgreSQL + Google Cloud Storage, at +`github.com/TheJacksonLaboratory/jabs-hub`). The desktop GUI is a **caching client** — it pulls +what it needs from Hub / GCS into a local **cache directory**, works against the cache, and pushes +changes back. There is **no hybrid**: a Hub-backed project never has authoritative local-only +media. Local-only (non-Hub) projects are unchanged. + +- **Videos live in a shared library.** Videos — external/uploaded files *or* JABS-2.0 device + recordings — are uploaded into a Hub **video library**, decoupled from any recording session. A + project references library videos **many-to-many**: one stored copy can back multiple projects. +- **Source of truth = cloud; local = cache.** One uniform pull → cache → push model for every + *source* artifact (video, pose, annotations, metadata). **Derived** artifacts (features, + predictions, classifiers) are recomputed locally from the cached pose and stay local (§4.9). +- **Offline-first.** The cache makes a project fully usable offline; edits queue and sync on + reconnect. +- **Primary driver.** Replace the "zip up `jabs/annotations/` and email it" label-sharing workflow + with automatic, versioned sync (§2.3). Because the exact pose file also comes from Hub, the + identity-alignment footgun (labels are keyed by identity, which the pose file defines) is + **eliminated**, not merely flagged. + +**This is a two-part plan**, split across the two repositories it touches: + +- **Client integration** — *this document* (`jabs-behavior-classifier`): OIDC auth, the Hub + client, the caching layer (media resolver + annotation cache), sync engine, opt-in/open flows, + and playback of Hub-hosted media. +- **Hub backend** — `jabs-hub` repo, `docs/specs/0006-jabs-project-backend-design.md`: the video + library + decoupled upload, the `projects` / `project_videos` (many-to-many) / `annotations` + data model, the REST API, membership/authorization, and media object storage. + +### Phasing (cloud-native, library-first build order) + +| Phase | Title | Delivers | +|------|-------|----------| +| **0** | Foundations | OIDC auth in the client + `HubClient`; the client cache/abstraction seams (annotation store + media resolver) landed as a pure refactor with local-only behavior unchanged. (Hub: auth on new routes, base scaffolding.) | +| **1** | Video library + media | Client: a Hub-aware media resolver + local media cache + lazy hydration + playback prefetch — download/cache to **open and play** a Hub project's videos. (Uploading into the library is a web-UI / `jabs-cli` concern.) (Hub: `videos` table, decoupled upload, list/search, content-addressed storage, signed download.) Deliverable: the GUI opens a Hub project and plays its videos. | +| **2** | Hub-backed projects | Client: **open** a cloud project (created in the web UI) referencing library videos; the local project dir is a cache; project-settings sync. (Hub: `projects` + `project_members` + `project_videos` join + metadata.) | +| **3** | Annotations (label sharing) | Client: annotation cache + sync engine + offline outbox + conflict handling; pose consistency guaranteed. (Hub: `annotations` + history + optimistic-concurrency contract + behavior index.) Delivers the label-sharing payoff. | +| **Post-MVP** | Collaboration + ML | Web preview playback + prediction-track overlay (library browse itself is part of the initial web UI), classifier registry/model cards, real-time collaboration, normalized fine-grained labels. | + +Phase 0 lands the seams first so they are independently reviewable and de-risk the rest (mirrors +the 0001 Parquet feature-cache implementation plan, which extracted the I/O boundary +before changing behavior). + +**Relationship to prior work:** this supersedes the older *"JABS Hub"* Word design doc (the +device/recording/processing platform already exists) **and** an earlier iteration of this plan that +migrated annotations first and left media local. The cloud-native model was chosen deliberately +over that piecemeal approach: the video library needs cloud media anyway, and full cloud storage +makes the label-sharing use case correct (shared pose ⇒ aligned identities) rather than merely +warned. + +--- + +## 2. Goals, Non-Goals, and Motivating Use Cases + +### 2.1 Goals + +1. **Cloud-native Hub-backed projects.** A Hub-backed project stores its video, pose, annotations, + and metadata in Hub; the GUI caches locally and stays fully usable offline. Local-only projects + are unaffected. +2. **Video library as the media home.** The GUI pulls library videos (and pose) down for + playback/labeling/training. Uploading into the library is a web-UI / `jabs-cli` task, not a + desktop-GUI flow. +3. **Automatic annotation sync** with version history, attribution, and conflict handling — + replacing the manual zip hand-off. +4. **Uniform caching model.** One pull/cache/push mechanism for every source artifact; derived + artifacts recomputed locally from the cached pose. +5. **Single source of authority.** Hub is authoritative online; the client reconciles the cache + against it, resolving the "which copy is current" ambiguity. + +### 2.2 Non-Goals + +- Replacing the desktop GUI with a web labeling client. The GUI stays the frame-accurate labeling + surface; the JABS Hub web UI (to be developed) creates/manages projects and the video library but + does not do frame-accurate labeling. +- **Creating or managing Hub projects in the desktop GUI** (create a project, add library videos, + share) — that happens in the JABS Hub web UI. The desktop GUI **opens** an existing Hub project by + its identifier (§4.7). +- Storing **derived** artifacts (features/predictions/classifiers) in Hub for the MVP — they are + recomputed locally from the cached pose (§4.9). An **optional cloud cache for features** (the + expensive one) and a classifier registry for predictions/classifiers are post-MVP (§10). +- Real-time collaborative cursors / live presence (post-MVP). +- True frame streaming of remote video (playback is download-to-cache + prefetch; §4.6). + +### 2.3 Motivating use cases + +1. **Asymmetric label sharing (primary driver; needs Phase 3 annotation sync *and* project sharing, D19).** A behavior expert labels + videos; a colleague trains/evaluates classifiers from those labels. Today the expert zips + `/jabs/annotations/` and sends it — manual, overwrites local work, no history, and no + guarantee both sides use the *same* pose file. In a cloud project both users open the same Hub + project; labels sync automatically with history and attribution, and because the **pose file is + pulled from Hub too**, the collaborator's labels always align to the correct identities. No zip, + no overwrite, no pose-mismatch footgun. +2. **Create a project in the web UI, open it in the GUI.** In the JABS Hub web UI a user creates a + project and adds videos to it from the library (JABS recordings or uploaded external files), then + opens it in the desktop GUI (§4.7) — the media hydrates on demand. +3. **Work from any machine / HPC** with no manual file copying — the cache hydrates from Hub. + +### 2.4 Unobtrusiveness for non-Hub users (hard requirement) + +Many JABS users do not have access to (or do not use) JABS Hub. Hub integration MUST be invisible +to them: + +- **No startup cost.** The app launches and every local-project workflow behaves exactly as today — + no network calls, no auth prompts, no "sign in to Hub" nags, no added latency. Hub code paths are + **lazy-imported** and never execute until the user invokes a Hub action. +- **One unobtrusive entry point.** A single **File → "Open Project from JABS Hub"** menu item + (§4.7); no toolbars, banners, or modal prompts intrude on local use. Local project open/create + never touches Hub. +- **Opt-in auth.** OIDC login is triggered only by a Hub action, never at startup; credentials live + in the OS keyring, never in project files. +- **Graceful absence.** If the Hub client dependency or connectivity is unavailable, only Hub + actions are affected — local projects are untouched, with a clear message rather than a crash. + +A regression test asserts that launching the app and opening a local project make **zero** Hub +imports or network calls (§7). + +--- + +## 3. Current State — client seams we build on + +### 3.1 Video/pose resolution seams (now core, not Phase 2) + +- Video↔pose pairing is name-derived: `NAME.mp4` ↔ `NAME_pose_est_v{N}.h5` (`_POSE_SUFFIX_RE`, + `packages/jabs-core/.../utilities.py:69`; `get_pose_path`, + `src/jabs/pose_estimation/__init__.py:40`). +- **Single video path resolver:** `VideoManager.video_path(video_file)` → + `Path(video_dir, video_file)` (`src/jabs/project/video_manager.py:251`). **This is the seam a + Hub-aware resolver overrides** to download-on-demand into the cache. +- **Single video open point:** `VideoReader.__init__` → `cv2.VideoCapture(str(path))` + (`src/jabs/video_reader/video_reader.py:19`). Path-only; needs random-access seeking → the + client must **download the full file to the cache before opening** (no true streaming). +- **`video_dir` / `pose_dir` are already decoupled** from the project directory (`ProjectPaths`, + `src/jabs/project/project_paths.py:18`; `Project.__init__` accepts them, `project.py:189`). This + is exactly the plumbing to point at a cache root. +- **Pose loading needs the full local file** (hashed end-to-end via `hash_file`, + `utilities.py:39`; h5py random access). No lazy/partial read → download-to-cache. +- **Feature/prediction caches key on name + validate on pose hash** (blake2b), not on path + (`src/jabs/feature_extraction/features.py:178`; `packages/jabs-io/.../feature_cache/base.py:71`; + `src/jabs/project/prediction_manager.py:167`). **Consequence:** a byte-identical pose pulled from + Hub keeps derived caches valid — the basis for keeping derived artifacts local (§4.9). +- **Project open touches every video/pose file** (`_validate_pose_files`, `video_manager.py:238`; + scan workers, `parallel_workers.py:175`) — must become **manifest-driven + lazy** so opening a + cloud project does not download everything up front (§4.6). + +### 3.2 Annotation seams + +- **Writes funnel through one method:** `Project.save_annotations(annotations, pose)` + (`src/jabs/project/project.py:618`) — atomic temp-file `replace()` into + `jabs/annotations/