Skip to content

README.en.md

Latest commit

 

History

History
540 lines (411 loc) · 32.9 KB

README.en.md

File metadata and controls

540 lines (411 loc) · 32.9 KB
Raw
RareLink evidence cockpit running in a local browser

RareLink

Keep data inside hospitals. Collaborate across sites. Verify every claim.

A trusted federated research operating system for rare disease and small-cohort imaging

中文 · English

📘 Project report · Research operations plane · Signed evidence package · Quick start · Three-Spark deployment · Itemized status and limits

NVIDIA DGX Spark NVIDIA FLARE MONAI Step 3.7 RareLink CI 406 tests passed Apache-2.0

Research-use engineering software; not diagnostic or treatment software. Current hardware evidence uses three logical sites on one real DGX Spark, plus a Spark–Mac mTLS exercise. The repository implements the protocol for three independent Sparks, but L1/L2 software evidence is never presented as L3/L4 physical-hospital or clinical validation.

RareLink at a glance

RareLink is neither three scripts hosted on one machine nor an LLM with access to patient records. It brings research governance, hospital-local compute, federated training, governed Agents, and signed evidence release into one operating system.

Product plane Problem solved Implemented capability
Research Operations Operate multiple studies, sites, and model versions over time Study switching, site admission/pause/withdrawal, model registry, evidence verification/release/revocation
Physical Federation Control Make the web console control real Sparks Independent Site Agents, external Job ID, SSE, abort/retry/resume, strict 3/3
Hospital-local Data Keep data local while proving readiness NIfTI/BIDS, DICOM-header gate, four-modal/geometry/label QC, MONAI cache, de-identified data digest
Federated Security Federation is not automatically secure mTLS, update clipping/anomaly checks, DP-SGD, privacy budgets, late/replay rejection, model signatures
Agent Governance Let models assist without overreach Five roles, Step 3.7/optional local TensorRT-LLM, bidirectional policy gates, human approval, red-team tests
Evidence & Release Make results independently reviewable and revocable Evidence Package v2, offline verifier, audit chain, SBOM, Cosign, ARM64 offline bundle

What the repository can prove today

Evidence Result Level and boundary
Software quality gates 406 Python tests; Ruff, production web build, release contract, and migration round trip pass L1/L2, not hospital acceptance
Independent-process protocol Three Site Agents, separate stores, signed heartbeats, exact 3/3 contract L2, not three physical devices
Public-imaging run on DGX Spark 24 MSD Task01 cases, four-modal QC, CUDA training, 3/3 FedAvg, persisted global model One Spark, three logical sites; not clinical performance
Safety and stability 25/25 seeded strategy runs, 26/26 Agent gates, DP accounting, mTLS positive/negative controls Engineering evidence, not certification
Remote CI All five GitHub Actions jobs passed Python 3.11/3.12, web, and both general-purpose image architectures

Five-minute start

No medical images, model weights, certificates, or API keys are required:

git clone https://github.com/dingyucanada/RareLink.git
cd RareLink
bash scripts/review_demo.sh

The script starts FastAPI and React and verifies public, de-identified engineering receipts. Missing hardware evidence is shown as a labelled snapshot or NOT CLAIMED; it is never fabricated.

Live product interface

Expand the full RareLink research and evidence cockpit
Full RareLink product cockpit

Contents

Why RareLink

Rare-disease, paediatric-tumour, and other small-cohort imaging research rarely fails because a model is missing. It fails because source data cannot be pooled, sites differ, study plans drift, experiments are hard to replay, an average metric can hide a weak site, and language models must not cross clinical-data boundaries.

RareLink joins protocol design, site feasibility, experiment contracts, local training, federated aggregation, privacy review, and research reporting into a controlled loop: data stays with the department, models train locally, only approved updates and aggregate metrics cross sites, and every consequential step leaves verifiable evidence.

Research challenge Product response
Source MRI, labels, and patient fields cannot be pooled NIfTI and labels remain local; policy gates reject source images, identifiers, DICOM UIDs, secrets, paths, and small-cell fields from outbound flows
One average score can hide site risk Mean Dice, weakest-site Dice, site spread, and HD95 are fixed in the contract and surfaced together
Processes are coordinated in documents and conversations A state machine, locked contract, hashes, audit ledger, and one-click verification turn the process into a product
Agents can overreach or leave no trace Five role-specific Agents see de-identified protocols and aggregates only; human approval, schemas, and bidirectional gates constrain them
“Federated” is often confused with “automatically compliant” Security communication, DP accounting, minimum-cell policy, failed configurations, and claim boundaries belong in the evidence package

Built for the people doing the work

User What RareLink helps them do
Department research teams Draft studies, assess site feasibility, inspect training/aggregation state, and export traceable research evidence
Multi-centre coordinators Fix endpoints, budgets, and egress policy; compare sites; manage approvals and audit trails
AI and platform teams Deploy MONAI, NVIDIA FLARE, control-plane services, and an optional local-LLM route on DGX Spark
Partners and governance teams Inspect system boundaries, evidence provenance, privacy constraints, and pilot prerequisites rather than receiving a single performance number

From a research question to an evidence package

RareLink is not a standalone “train” button. It is a recoverable, auditable research workflow:

research question
  → structured protocol
  → site feasibility (minimum necessary aggregates only)
  → experiment contract + human lock
  → local training / federated aggregation
  → metrics, model and run receipt in ledger
  → statistics and privacy review
  → research report and evidence package
  1. Define the protocol. A researcher describes a question, cohort boundary, and objective; the Research Director structures it without seeing case-level images.
  2. Confirm feasibility. Sites return only aggregates that meet the minimum-cell policy.
  3. Lock the contract. Splits, strategies, rounds, primary endpoints, weakest-site metrics, and release limits are fixed by a human principal investigator.
  4. Compute locally. DGX Spark runs MONAI/PyTorch 3D training; NVIDIA FLARE coordinates approved updates and aggregation.
  5. Interpret without overclaiming. Statistical and Privacy Agents work from aggregate evidence and can flag risk or block unsafe release.
  6. Deliver an inspectable package. Hashes, metrics, model paths, policy, and stated limitations travel together.

Research operations plane

A successful training run is not an operable product. The Research Operations Plane turns studies, sites, models, and evidence into queryable, governed, and revocable objects.

flowchart LR
    S["Study Registry\norganisation · study · revision"] --> M["Site Membership\ninvite · admit · pause · withdraw"]
    M --> J["Physical FLARE Job\nexternal Job ID · round · 3/3"]
    J --> V["Model Registry\nversion · digest · metrics · signature"]
    J --> E["Evidence Registry\nDP · security · dual approval · signature"]
    E -->|"same study, digest, and tier"| V
    E -->|"revocation cascade"| X["Model REVOKED"]
    V -->|"independent approval and release"| R["Research Release"]
Loading

Governed site membership

INVITED → ACTIVE ↔ PAUSED → WITHDRAWN

A site cannot become active without data-use approval, certificate binding, and a de-identified dataset fingerprint. Withdrawal is terminal; old authority cannot be silently restored through configuration.

Evidence-bound model lifecycle

CANDIDATE → STATISTICAL_REVIEW → SECURITY_REVIEW
          → APPROVED → RELEASED → REVOKED

The registry stores semantic version, source FLARE Job, artifact SHA-256, validation tier, aggregate metrics, key fingerprint, and evidence binding—not the model binary. Creator, approval reviewer, and release approver are separate actors.

Formal evidence accepts only L3 physical or L4 hospital validation with exact quorum, privacy and security gates, distinct approvals, sensitive-content scanning, and package/model signatures. Revoking evidence automatically revokes every bound model. See the Research Operations Plane specification.

Product experience and capabilities

The frontend explicitly separates an interactive research-workflow sandbox from persisted hardware evidence. The former helps teams understand protocol, contract, Agent, and approval states; the latter verifies training, aggregation, and security claims.

Research workflow and boundaries Evidence cockpit
RareLink workflow overview Live RareLink evidence cockpit

Study protocol, contracts, and audit ledger

  • Turn a research question into a structured protocol with approval, version, and state transitions.
  • Lock strategy, rounds, endpoints, weakest-site metrics, and egress limits into an experiment contract.
  • Persist training jobs, logs, aggregates, model paths, failures, and retries in a traceable ledger.
  • Retry failed work without silently overwriting evidence or creating a duplicate experiment.

Federated research designed for site differences

  • Compare Local, FedAvg, FedProx, strict SVT, and sample-level DP-SGD.
  • Evaluate mean Dice, weakest-site Dice, site variance, and HD95 together.
  • Serialize 3D workloads in the one-Spark prototype with a unified-memory guard.
  • In a real deployment, each department runs an independent Spark Client and the coordinator receives approved updates and aggregates only.

Evidence cockpit and one-click verification

  • Read persisted public-data run receipts with 3/3 aggregation, global-model, site-metric, and boundary information.
  • Verify local evidence hashes and expand site-level Dice, HD95, and timing.
  • Start a replay package without downloading medical images, weights, certificates, or API keys.
  • Surface “endpoint online,” “receipt captured,” and “independently verified” separately; missing evidence is NOT CLAIMED, never fabricated.

Research safety is a product feature

Input gates reject source images, identifiers, DICOM UIDs, paths, and secrets. Output gates block diagnostic instructions, patient-data requests, clinical overclaims, and unapproved contract escalation. Privacy review can block reporting, but cannot relax deterministic egress policy.

RareLink engineering evidence scorecard

System architecture and data boundaries

RareLink relies on mature frameworks for commodity functions. Project-specific code concentrates on the research state machine, experiment contracts, data-egress policy, evidence linkage, and the Spark/FLARE adapter—keeping safety-critical logic readable, testable, and auditable.

flowchart TB
    U["Researchers / department teams"] --> W["React evidence cockpit"]
    W --> A["FastAPI control plane"]
    A --> L["PostgreSQL / Alembic\nstudies · sites · jobs · models · evidence · audit"]
    A --> Q["Research Operations Plane\nmulti-study · model registry · evidence lifecycle"]
    A --> P["Protocol, contract, and policy state machine"]
    A --> G["Input / output policy gates"]
    A --> R["FederationRunner"]
    R --> M["Mock Runner\nlocal development, always labelled mock"]
    R --> F["NVIDIA FLARE\nFedAvg / FedProx / Client API"]
    F --> S1["Site A: local MONAI / PyTorch"]
    F --> S2["Site B: local MONAI / PyTorch"]
    F --> S3["Site C: local MONAI / PyTorch"]
    G --> T["Step 3.7\nde-identified protocols and aggregates"]
    G --> N["Optional TensorRT-LLM\nprivate Spark endpoint"]
    T --> O["Five-role Agent Team"]
    N --> O
    O --> A
    R --> E["Evidence Package v2\noffline verification · model signing · revocation"]
    E --> Q
Loading
Boundary System rule Purpose
Data boundary NIfTI, labels, patient fields, and DICOM UIDs stay on site Keep source case material out of control-plane and Agent flows
Compute boundary Imaging training runs locally on DGX Spark; federation coordinates approved updates Keep heavy computation near the data and reduce centralized exposure
Language-model boundary Step 3.7/local LLMs consume de-identified protocol and aggregate context only Let Agents assist the research process without reading imaging or credentials

Multi-Agent collaboration and model strategy

RareLink does not let a generic chatbot decide a study. It divides work into verifiable, blockable, and recoverable roles.

Agent role Permitted input Output Cannot do
Research Director De-identified question and cohort description Structured research protocol Access images, labels, or case fields
Experiment Designer Protocol and feasibility aggregates Local/FedAvg/FedProx comparison contract Lock the contract or change endpoints alone
Statistical Reviewer Verified aggregate metrics Weakest-site risk, metric interpretation, limits Turn small, one-off results into medical conclusions
Privacy Reviewer Report draft and policy state Release/block recommendation Relax data-egress policy
Research Writer Approved evidence and limits Traceable research narrative Generate diagnostic advice or invent a run result
Route Best for Boundary and fallback
step_remote Step 3.7-assisted protocol, statistics, and writing Sends policy-filtered text/aggregates only; can fall back when the API is unavailable
spark_local TensorRT-LLM on a private Spark endpoint Aggregates only; requires a real GPU and content-free receipt before being described as verified
template Offline demos, tests, and no-key environments Deterministic template Agent keeps the workflow stable and testable; never presented as model inference

Step 3.7 integration is not merely a configuration flag: every successful live call that passes JSON-schema validation and the output safety gate produces a local metadata-only receipt. It contains no prompt, completion text, image, case field, or credential. The evidence cockpit shows STEP 3.7 AGENT RUNTIME · VERIFIED only when such a receipt exists; see the deployment guide for the exact commands and boundary.

The TensorRT-LLM adapter, deployment scripts, receipt tooling, red-team harness, and concurrency benchmark exist. Until a real Spark model call, red-team run, and benchmark are captured, RareLink does not claim measured local-LLM performance or medical capability.

DGX Spark and NVIDIA foundation

RareLink treats DGX Spark as a trusted department-side compute unit, not merely a web host. It can host 3D training, a federated Client, control-plane services, evidence delivery, and an optional local-Agent boundary. For medical research, this full-stack deployment near the data is more appropriate than moving large images to a remote service.

Technology Role in RareLink Current implementation or evidence
NVIDIA DGX Spark GB10 / ARM64 local compute and runtime boundary CUDA, MONAI 3D, FLARE, API/web ran on hardware; one MSD three-site round completed in about 69 s
CUDA + PyTorch Tensor compute, AMP, and training runtime MONAI training and federated client work on Spark
NVIDIA FLARE FedAvg/FedProx, Client API, mTLS, orchestration Three logical-site aggregation; Spark–Mac mTLS registration, reconnect, and invalid-identity rejection exercise
MONAI NIfTI, 3D SegResNet, transforms, Dice/HD95 Synthetic controls and public MSD Task01 engineering verification
TensorRT-LLM (optional) Private OpenAI-compatible Agent endpoint on Spark Code, deployment/verifier, 26 local-gateway red-team cases, and 1/2/4 concurrency tools are delivered; no receipt means NOT CLAIMED
Step 3.7 Policy-constrained protocol, experiment design, statistics/privacy review, and writing Receives de-identified text and aggregates only; deterministic template fallback without a key

This division addresses three deployment facts at once: 3D imaging needs local compute, institutions cannot casually move source data, and research conclusions must remain traceable. NVIDIA FLARE alone does not guarantee compliance; RareLink adds policy, identity, audit, and DP-SGD controls around it.

Engineering evidence and claim boundaries

Public real-imaging engineering verification

RareLink completed an engineering-compatibility verification on public MSD Task01_BrainTumour data using an NVIDIA DGX Spark GB10 (ARM64, CUDA 13, PyTorch 2.10.0+cu130, MONAI 1.6.0, NVIDIA FLARE 2.7.2). The archive was downloaded directly by Spark and checked against its published MD5 and a local SHA-256. The repository stores no images, labels, case IDs, case-level paths, or weights.

After geometry checks, 24 four-modality 3D MRIs were deterministically divided by tumour-voxel quantiles into three logical sites of eight cases. The MSD 0/1/2/3 labels were mapped to the project’s 0/1/2 contract; depth 155 was padded to 156 using DivisiblePadd(k=4).

Run Training configuration Verifiable result
Single-site CUDA smoke test site-a: 7 train / 1 validation case, 1 epoch Loss 1.818046; mean foreground Dice 0.008702; HD95 117.379684; 6.7476 s; peak GPU memory 5240.349 MiB
Three-logical-site FedAvg 8 cases/site, 1 local epoch, 1 federation round 3/3 client updates aggregated with global model persisted; 69.0084 s end-to-end; peak GPU memory 5240.349 MiB
Site Dice HD95 Training loss Local duration
site-a 0.012345 119.771957 1.657893 6.6302 s
site-b 0.040765 107.582993 1.653218 6.3943 s
site-c 0.071763 79.886047 1.649940 6.7046 s
Aggregate observation 0.041624 102.413666 — 69.0084 s

Stability, privacy, and safety checks

Check Method Current result Boundary that remains
Stability 5 random seeds × Local / FedAvg / FedProx / strict SVT / DP-SGD, 3 rounds 25/25 combinations completed Synthetic data/logical sites only; not medical statistical inference
Sample-level DP-SGD Opacus clipping, Gaussian noise, Poisson sampling, RDP accounting Conservative three-round ε=6.076881, δ=1e-5 Local training steps only; not end-to-end, user-level, or hospital-level DP
Secure federation Spark–Mac mTLS registration, reconnect, invalid-identity rejection Positive and negative controls recorded Not a hospital production network, identity system, or complete penetration test
Agent safety 12 input + 14 output deterministic gateway cases 26/26 passed Tests policy gateways; not general model-safety certification

These results do demonstrate real four-modality NIfTI intake, CUDA training, FLARE aggregation, global-model persistence, and controlled Agent workflows. They do not demonstrate clinical efficacy, strategy superiority, hospital-WAN operation, or paediatric rare-disease performance. Explicit boundaries are a core part of a trustworthy research platform.

Deployment and quick start

Three deployment layers

Layer Purpose Deployment
Local development / demo Explore the workflow and evidence cockpit React + FastAPI + SQLite; deterministic template Agents and labelled evidence snapshots are available
Single-Spark engineering validation Validate CUDA, MONAI, FLARE, policy, and evidence chain One DGX Spark runs three logical sites and serializes work to protect unified memory
Real multi-hospital research pilot Independent local computing without exporting source data One Spark Client per hospital + certificate-based FLARE + local audit; requires IRB, data-use, and security review

Three-physical-site control plane

The repository now implements a P0 control loop for independently deployed devices instead of hard-coding three departments into one process. Each Spark runs its own Site Agent and SQLite state store, checks GPU memory and temperature, CPU, memory, disk, MONAI, NVIDIA FLARE, certificate identity/CA chain/offline CRL, and its local manifest, and emits only an HMAC-signed, patient-free heartbeat. The coordinator persists the real NVIDIA FLARE Job ID and supports human-approved submission, status reconciliation, abort, retry, resume, fixed 3/3 quorum, and SHA-256 verification of a completed global model. The Site Agent can control the local FLARE Client lifecycle through a fixed-unit, shell-free systemd adapter, with signed-checkpoint gates for safe pause/recovery. The web console receives resumable SSE events and shows site, round, update, artifact-integrity, and review-gate state without polling. Before training, the hospital data layer validates four-modality NIfTI geometry and the label contract and creates a path-free, case-ID-free dataset fingerprint. A version change at any site automatically invalidates the old job.

flowchart LR
    A["Hospital A Spark\nSite Agent + FLARE Client"] -->|"signed heartbeat / mTLS update"| C["RareLink Coordinator\nFastAPI + FLARE Admin"]
    B["Hospital B Spark\nSite Agent + FLARE Client"] -->|"signed heartbeat / mTLS update"| C
    D["Hospital C Spark\nSite Agent + FLARE Client"] -->|"signed heartbeat / mTLS update"| C
    C --> E["Physical operations view\nJob ID · Round · 3/3 Quorum · Model Hash"]
Loading

Before three devices are available, exercise the same protocol with three independent operating-system processes:

make physical-control-smoke

This verifies distinct Site IDs and local stores, signed heartbeats, coordinator registration, and a 3/3 contract. It is explicitly isolated-integration: it runs no medical training and is not presented as three-Spark evidence. See the physical deployment guide for services, secrets, and field operations, and the formal engineering plan for the WBS, state machines, threat model, and acceptance levels; and the physical-control audit design for the chained events, HMAC key gate, public/protected API boundary, verification method, and explicit production limitations; and the physical identity and RBAC design for offline OIDC validation, the five-role/nine-permission matrix, and remaining hospital IAM work; and the physical contract dual-approval design for canonical contract locking, distinct-subject approval, revalidation, and remaining approval-lifecycle work. The hospital-local NIfTI specification defines the manifest, validation, and version-invalidation contract.

Run the complete software and isolated-integration gate:

make p0-p1-acceptance

The current baseline is 406 Python tests plus Ruff, the release-engineering contract, a production web build, three independent control-plane processes, a seven-scenario fault matrix, PostgreSQL production-compose validation, and an Alembic migration round trip. This is L1/L2 evidence, not a three-device or hospital-production claim. P1-S06 selects NVIDIA FLARE FedAvgHERecipe with TenSEAL as the secure-aggregation candidate. It remains fail-closed because TenSEAL and the three-device key/performance evidence are absent:

make secure-aggregation-assessment

When three devices are available, the read-only field tool checks the real Site Agents, NVFLARE Job ID, 3/3 quorum, model signature, and audit chain without exporting credentials or endpoint URLs. A completed study can then be released as a Research Evidence Package v2. It binds two distinct approvals, three per-site data cards, strict 3/3 completion receipts, aggregate metrics, the DP ledger, four Agent/ART gates, complete audit linkage, and the signed global model to one canonical manifest. A standalone verifier is embedded and covered by the package Ed25519 signature; any missing site, privacy overrun, failed security gate, broken chain, patient field, case identifier, path, or secret fails closed. The isolated fault matrix checks outage recovery, Agent restart, GPU/disk/certificate gates, and duplicate/late updates:

make fault-injection-matrix

One-command experience

No medical images, weights, certificates, or API keys are needed:

git clone https://github.com/dingyucanada/RareLink.git
cd RareLink
bash scripts/review_demo.sh

The terminal prints a local URL. The package checks 25/25 repeated experiments, DP-SGD accounting, Spark–Mac mTLS negative controls, and 26/26 Agent gateway cases. When local artifacts are missing, it writes only clearly labelled de-identified demonstration snapshots; it never presents them as new experiments.

Local development

cp .env.example .env
python3 -m venv .venv
. .venv/bin/activate
make install
make install-web
make dev-api

In a second terminal:

make dev-web

Open http://localhost:5173; API docs are at http://localhost:8000/docs. Without STEP_API_KEY, deterministic template Agents still run the controlled workflow. Keep all secrets in local .env files.

Optional public-imaging engineering run on Spark

Download public MSD Task01 directly on the Spark node; do not transfer large files through SSH/SCP:

python scripts/prepare_msd_brain_tumour.py \
  --data-root data/raw/msd-task01 \
  --output data/runtime/msd-brain-tumour-v1 \
  --cases-per-site 8

python scripts/run_nvflare_simulation.py \
  --manifest data/runtime/msd-brain-tumour-v1/manifest.json \
  --strategy fedavg --rounds 1 --local-epochs 1 \
  --workspace artifacts/msd-fedavg

Before running, confirm aarch64, nvidia-smi, Python 3.11+, free disk, and a safe unified-memory margin. Use the NVIDIA PyTorch container or the repository Spark bootstrap script where appropriate.

Roadmap, references, and responsible use

Next milestones

Stage Goal Prerequisites
External engineering validation Repeat the workflow with authorised paediatric public data such as BraTS-PEDs Data-use policy, preprocessing, citation, and transparent reporting limits
Real multi-hospital pilot Deploy the existing Site Agent and control API to three Sparks and complete Level 2 failure/recovery acceptance IRB, data-use agreement, security review, network, certificate, and identity readiness
Clinical research collaboration PACS/FHIR integration, governance, cross-site feasibility, evidence packages Institutional partners, independent clinical validation, regulatory pathway
Local-Agent upgrade Capture real TensorRT-LLM receipts, concurrency data, and security red-team evidence Available local model and compliant inference resources

Further reading and primary sources

DGX Spark Hackathon note

RareLink was built and open-sourced during the DGX Spark Hackathon. Its purpose is to show, through runnable software, public engineering evidence, and explicit limits, how DGX Spark, NVIDIA FLARE, MONAI, and Step 3.7 can work together for multi-centre research where data remains on site. It is not a substitute for clinical validation or a real hospital-deployment claim.

Safety and responsible use

Never commit API keys, passwords, patient images, DICOM identifiers, source manifests, or identifiable clinical fields. Follow the licence, citation, and access policy of every public dataset. Any clinical or real multi-hospital deployment requires institutional approval, data-use agreements, security review, and independent validation.

License

Apache-2.0