RNA-seq Analysis Pipeline

A config-driven, HPC-batch-scheduled RNA-seq pipeline: raw paired-end FASTQ through adapter trimming, HISAT2 alignment, BAM processing/QC, and featureCounts gene-level quantification, with a merged, sample-metadata-aware count matrix as the final output.

This repository is a sanitized snapshot of a working research pipeline. Absolute paths and the originating HPC username have been replaced with placeholders, and directories holding raw sequencing data (FASTQ/BAM) and project-specific sample metadata have been excluded.

Architecture

Scripts/
├── config.py                      # Single source of truth: paths, genome build, read-strategy,
│                                   # trimming/dedup flags, sample-naming patterns — every step
│                                   # reads from here instead of hardcoding values.
├── step_0a_setup.py                # Directory scaffolding
├── step_0b_rename_fastq_files.py   # Strip sequencing-core suffixes from raw filenames
├── step_0c_prepare_samples_file.py # Build samples.txt / sample_metadata.txt from filename patterns
├── step_0d_bam_to_fastq.py         # Optional BAM → FASTQ conversion (re-analysis from public BAMs)
├── step_0f_fastqc.py               # Per-sample FastQC
├── step_1_trimmomatic.py           # Adapter/quality trimming
├── step_2_align_hisat2.py          # Spliced alignment against the reference transcriptome
├── step_3_process_bam.py           # Sort/filter/index BAMs
├── step_3qc.py / step_3qc_report.py  # Alignment QC + summary report
├── step_4_count_reads.py           # Per-sample featureCounts submission
├── step_5_summarize_reads.py       # Merge per-sample counts into one gene x sample matrix,
│                                   # renamed via sample metadata
├── prepare_samples_encode.py       # Build a sample sheet from ENCODE-style inputs
├── prepare_samples_vs_input.py     # Build sample/control pairings
├── geneExpression_heatmap.R        # Expression heatmap visualization
├── generate_genome_indexes.sh      # STAR/HISAT2 genome index generation
└── utils/ldsample.py               # Shared sample-metadata loader used by every step

Design principles

• Paths are derived from config.py.
• Every step is per-sample and submits its own HPC batch job (bsub), writing its own .batch/.log/.error files so a failed sample can be re-run in isolation.
• utils/ldsample.py centralizes sample handling: it reads Metadata/samples.txt, resolves each sample name to its FASTQ/BAM source file, and tracks sample/input-control pairings.

Usage

1. Create the conda environment: conda env create -f environment.yml
2. Edit Scripts/config.py: set USERNAME, PROJECT_NAME, reference genome/annotation paths, and sample-naming patterns (ANTIBODY_PATTERNS/GROUP_PATTERNS, used to auto-derive sample metadata from filenames).
3. Populate Metadata/samples.txt (sample name ↔ input-control pairing) and place raw FASTQ under Original_Data/fastq_files/.
4. Run steps in order, step_0a... → step_5_summarize_reads.py.

License

MIT — see LICENSE.