PrismSC: Multi-Modal Single-Cell Cohort Integration Pipeline

      /\
     /  \   P R I S M   S C
    /____\  [ rna | atac | wnn ]

PrismSC is a single-cell cohort processing workflow designed for reproducible and scalable analysis of joint scRNA-seq and scATAC-seq datasets. It integrates quality control, batch correction, automated cell annotation, lineage fate mapping, and interactive diagnostics HTML reporting.

---

Biological Context & Objectives

PrismSC is designed to answer key clinical questions:

1. Cellular Composition Analysis: Automatically identify and quantify cell populations across multiple patient cohorts under different clinical conditions (e.g., Healthy vs. Inflamed vs. Post-Treatment).
2. Multi-Modal Integration: Fuse transcriptomic and chromatin accessibility profiles using Weighted Nearest Neighbors (WNN) or probabilistic deep generative models (MultiVI, scVI).
3. Automated Annotation: Eliminate manual annotation bias by using pre-trained CellTypist immune classifiers.
4. Developmental Dynamics & Fate Mapping: Use CellRank 2 (GPCCA) and PAGA to map cell transition probabilities and fate commitments along differentiation trajectories (e.g., Monocyte subset maturation).
5. Interactive Diagnostics Reporting: Consolidate QC statistics, modality projections, cell proportions, and fate mapping drivers into a portable, researcher-ready HTML report.

---

Pipeline Architecture

                       Cohort Manifest (TSV)
                               │
                               ▼
        ┌──────────────────────────────────────────────┐
        │   Module 1: Preprocessing & Rust QC Summary  │
        │   • QC Filtering (Mito%, Gene counts)        │
        │   • Scrublet Doublet Removal                 │
        │   • Rust QC Aggregator (prism_qc_aggregator) │
        └──────────────────────┬───────────────────────┘
                               ▼
        ┌──────────────────────────────────────────────┐
        │   Module 2: Cohort Integration (Zarr DB)     │
        │   • scVI (transcriptomic VAE with 16-bit AMP)│
        │   • PeakVI (chromatin accessibility VAE)     │
        │   • MultiVI & WNN (joint multi-modal graph)  │
        └──────────────────────┬───────────────────────┘
                               ▼
        ┌──────────────────────────────────────────────┐
        │   Module 3: Automated Cell-Type Annotation   │
        │   • Pre-trained CellTypist Immune Models     │
        │   • Majority Voting Neighborhood Consensus   │
        │   • Dynamic Marker-Scoring Offline Fallback  │
        └──────────────────────┬───────────────────────┘
                               ▼
        ┌──────────────────────────────────────────────┐
        │   Module 4: Fate Mapping & Lineage Dynamics  │
        │   • Diffusion Pseudotime & PAGA Trajectories │
        │   • CellRank 2 Fate Absorption Probabilities │
        └──────────────────────┬───────────────────────┘
                               ▼
        ┌──────────────────────────────────────────────┐
        │   Module 5: Clinical Diagnostic Reporting    │
        │   • Self-contained HTML report with CSS      │
        │   • Embedded vector UMAPs & quality metrics  │
        └──────────────────────┬───────────────────────┘

---

Repository Structure

PrismSC/
├── config/
│   ├── config.yaml              # Pipeline hyperparameters and execution flags
│   ├── cohort_schema.json        # Manifest JSON schema validator
│   └── cohort_manifest.tsv      # Cohort metadata (patient_id, file paths, condition)
├── workflow/
│   ├── Snakefile                # Main Snakemake execution entrypoint
│   └── rules/
│       ├── preprocessing.smk    # QC and Rust aggregation rules
│       ├── integration.smk      # Multi-modal batch correction & VAE rules
│       ├── annotation.smk       # CellTypist cell typing rule
│       ├── dynamics.smk         # Trajectory & fate mapping rule
│       └── reporting.smk        # Clinical HTML dashboard generation rule
├── src/
│   └── prism_qc/                # High-performance Rust QC aggregation module
│       ├── Cargo.toml
│       └── src/main.rs          # Multi-sample JSON metrics aggregator binary
├── scripts/
│   ├── preprocess.py            # Single-sample QC, doublet removal, MuData slicing
│   ├── integrate.py             # Integration orchestrator & fallback diagnostics
│   ├── integrate_worker.py      # Subprocess trainer (scVI, PeakVI, MultiVI, WNN)
│   ├── annotate.py              # CellTypist annotation & marker-scoring fallback
│   ├── dynamics.py              # PAGA trajectories & CellRank 2 fate mapping
│   ├── generate_clinical_report.py # Jinja2 self-contained HTML report compiler
│   ├── log_utils.py             # CLI formatting, gradients, and log suppression
│   └── benchmark_and_test.py    # Validation suite (10x slicing, GPU benchmark, offline test)
├── envs/
│   └── clinical-sc-omics.yaml   # Conda environment definition
├── Dockerfile                   # Multi-stage container deployment definition
├── .gitignore                  # Git untracked pattern rules
└── README.md                    # Documentation (this file)

---

Getting Started

Prerequisites

• Python 3.10+ (Python 3.12 recommended for PyTorch CUDA compatibility)
• Rust & Cargo (for compiling the QC aggregation binary)

Fast Installation (uv — Recommended)

1. Create a Python 3.12 virtual environment and install dependencies:

   uv venv --python 3.12
   source .venv/bin/activate
   
   # CUDA GPU Acceleration (NVIDIA Ampere/Ada/Hopper)
   uv pip install torch --index-url https://download.pytorch.org/whl/cu124
   
   # Core bioinformatics & workflow suite
   uv pip install snakemake scanpy muon scrublet celltypist jinja2 scvi-tools cellrank

2. Compile the high-performance Rust QC aggregator:

   cargo build --release --manifest-path src/prism_qc/Cargo.toml
   chmod +x src/prism_qc/target/release/prism_qc_aggregator

3. Run the pipeline:

   snakemake --cores all

Alternative: Conda Installation

conda env create -f envs/clinical-sc-omics.yaml
conda activate clinical-sc-omics
cargo build --release --manifest-path src/prism_qc/Cargo.toml
snakemake --cores 8

Dry Run Verification

To verify the workflow DAG and configuration without executing compute jobs:

snakemake -n

---

Validation & Hardware Benchmarking

PrismSC includes a standalone test and benchmarking suite (scripts/benchmark_and_test.py) to audit pipeline stability and measure hardware acceleration:

python scripts/benchmark_and_test.py

This automated suite executes:

1. Real 10x Biological Slicing: Extracts a 1,000-cell raw binary slice from 10x multi-omics PBMC data to test I/O without synthetic data artifacts.
2. CPU vs. GPU Benchmark: Trains a deep generative variational autoencoder (scVI) to benchmark throughput across available CUDA devices and CPU cores.
3. Offline Fallback Validation: Simulates complete network isolation (blocking external socket requests) to verify that local marker-scoring heuristics (CD3D, CD14, CD68, MS4A1) gracefully recover cell annotation when external model repositories are unreachable.
4. End-to-End Snakemake Execution: Validates DAG generation, Rust QC consolidation, VAE integration, cell annotation, and HTML report compilation.

---

Docker Deployment

The pipeline is containerized using a lightweight Docker context (excluding large datasets via .dockerignore).

# Build the docker container image
docker build -t prismsc-pipeline .

# Execute on CPU (mapping results directory)
docker run -v $(pwd)/results:/app/results prismsc-pipeline snakemake --cores 8

# Execute with GPU Acceleration (requires NVIDIA Container Toolkit)
docker run --gpus all -v $(pwd)/results:/app/results prismsc-pipeline snakemake --cores 8

---

Cloud & HPC Orchestration

1. On-Premise Clusters (SLURM)

PrismSC includes a pre-configured Snakemake execution profile for SLURM cluster submission:

• Snakemake v8+ (Plugin Executor):

  pip install snakemake-executor-plugin-slurm
  snakemake --profile config/slurm

• Snakemake v7 (Legacy fallback):

  snakemake --cluster "sbatch --partition=standard --cpus-per-task={threads} --mem={resources.mem_mb}" -j 16

2. Cloud Orchestration (AWS Batch)

To run dynamically on AWS serverless compute resources, leverage the official AWS executor plugin to route files to S3 buckets:

pip install snakemake-executor-plugin-aws-batch snakemake-storage-plugin-s3

snakemake --executor aws-batch \
          --jobs 100 \
          --default-resources s3_bucket=my-single-cell-bucket

---

Tissue-Specific Configurations

To run the workflow on organs other than immune cells, modify params.annotation.model in config.yaml. The cell annotation script will automatically download the respective CellTypist neural net and route to matching lineage fallback marker panels if the model fails:

Organ	CellTypist Model	Fallback Lineage Markers
Immune (Default)	Immune_All_Low.pkl	T-cells, B-cells, Monocytes, NK, Granulocytes
Brain	Human_Dev_Brain.pkl	Neurons, Astrocytes, Oligodendrocytes, Microglia
Lung	Human_Lung_Atlas.pkl	Epithelial, Endothelial, Stromal, Immune
Kidney	Kidney_Biopsy.pkl	Podocytes, Proximal Tubule, Loop of Henle, Collecting Duct

GitHub Pages Deployment

The clinical diagnostics HTML report is set up to deploy automatically to GitHub Pages. To host and view the interactive, live report on the web:

1. Go to your PrismSC repository page on GitHub.
2. Click on the Settings tab at the top-right.
3. Select Pages on the left-hand navigation sidebar.
4. Under Build and deployment -> Source, select Deploy from a branch.
5. Set the branch to main and the target folder to /docs, then click Save.
6. GitHub will build the site; your interactive clinical dashboard will be live at: https://QntmSeer.github.io/PrismSC/

---

License

This project is licensed under the MIT License - see the LICENSE file for details.