H3N2 Flu Pipeline

A Nextflow-based computational pipeline for the structural and sequence analysis of H3N2 Influenza A sequences. This workflow integrates sequence parsing, homology modeling, structural stability evaluation, and protein language model (ESM-2) embeddings to generate structural graphs for downstream machine learning tasks.

🧬 Pipeline Overview

The pipeline executes the following steps:

1. FASTA Processing (SPLIT_PROTEIN_FASTA, SPLIT_NUCLEOTIDE_FASTA): Splits the protein and nucleotide multi-FASTA inputs into per-variant sequence files.
2. Homology Modeling (BUILD_HOMOLOGY_MODEL): Generates 3D structural models of the sequences using Modeller, based on a template structure (default: 4O5I.pdb).
3. Structure Refinement (CLEAN_PDB, REPAIR_PDB): Cleans the generated PDB files and repairs structures using FoldX.
4. Stability Analysis (CALC_STABILITY): Calculates the thermodynamic stability of the repaired structures using FoldX.
5. Distance Metrics (CALC_HAMMING): Calculates the Hamming distance between the sequences and the template structure.
6. Protein Language Modeling (RUN_ESM2): Generates embeddings and perplexity scores for each sequence using the ESM-2 model.
7. Graph Generation (GENERATE_GRAPH): Combines the 3D structural data (cleaned PDBs) and ESM-2 embeddings to construct graph representations (.graph.pt) suitable for Graph Neural Networks (GNNs).
8. CpG Analysis (CALC_CPG): Calculates CpG dinucleotide depletion ratios from each nucleotide sequence.
9. Feature Aggregation (AGGREGATE_RESULTS): Combines the Hamming distance, CpG ratio, ESM-2 perplexity, and FoldX stability score for each variant into aggregated_results.parquet.

🛠 Prerequisites

• Nextflow (21.04.0)
• Docker (The pipeline uses the vulcan:latest container image)
• DVC (Data Version Control) for managing large datasets/models.

🚀 Usage

1. Clone the repository:

   git clone https://github.com/thiyageshvenkat/h3n2-flu.git
   cd h3n2-flu

2. Prepare your data: Place the protein and nucleotide GISAID multi-FASTA files in the data/ directory and ensure the template PDB is available.

   • Protein input: data/gisaid_sequences/protein.fasta
   • Nucleotide input: data/gisaid_sequences/nucleotide.fasta
   • Template structure: data/4O5I_HA_AB.pdb

3. Optional: create .env from .env.example, fill in any values you need, and source it before running Nextflow

4. Run the pipeline:

   nextflow run main.nf \
   --protein data/gisaid_sequences/protein.fasta \
   --nucleotide data/gisaid_sequences/nucleotide.fasta \
   --template_pdb data/4O5I_HA_AB.pdb \
   --outdir results \
   -resume

📂 Output Directory Structure

The pipeline publishes results in the results/ directory:

• results/aggregated/: Aggregated numerical features in aggregated_results.parquet, suitable for analysis with pandas or DuckDB.
• results/duckdb/: Optional local DuckDB database files, such as h3n2.duckdb, for querying the aggregated Parquet output.
• results/split_protein_sequences/: Individual protein sequences split from the input multi-FASTA file (.split.fa). These files feed the modeling, Hamming-distance, and ESM-2 branches.
• results/split_nucleotide_sequences/: Individual nucleotide sequences split from the input multi-FASTA file (.split.fa). These files feed the CpG-analysis branch.
• results/models/: Raw homology models generated by Modeller (*_model.pdb).
• results/cleaned/: Cleaned homology models with unwanted PDB records removed (*_model_clean.pdb).
• results/repair/: FoldX-repaired structures used for stability calculations (*_Repair.pdb).
• results/stability/: FoldX thermodynamic stability results (*_ST.fxout).
• results/hamming/: Sequence-to-template Hamming-distance results (*_dist.txt).
• results/cpg/: CpG dinucleotide counts and depletion metrics for each nucleotide sequence (*_cpg.txt).
• results/esm2/: Per-sequence ESM-2 perplexity scores (*.perplexity) and residue embeddings (*.embeddings.pt).
• results/graphs/: Final PyTorch graph objects combining structural coordinates with ESM-2 features (*.graph.pt) for GNN input.

Older runs may also contain results/split_sequences/, the legacy location used before protein and nucleotide sequence outputs were separated. Nextflow work files and execution metadata remain in work/ and .nextflow/; they are not published pipeline results.

🔍 Error Monitoring (Optional)

This pipeline optionally integrates Sentry for easy remote error tracking.

To enable it, set SENTRY_DSN in .env and source that file before launching Nextflow. See .env.example for the expected format.

If SENTRY_DSN is not exported into the environment, Sentry is automatically disabled and the pipeline will run normally.

📂 Data Version Control (DVC)

After running the pipeline, you may be interested in using DVC for the outputs.

Push results to Google Drive:

dvc push -v
Check remote config
Get-Content .dvc\config
Get-Content .dvc\config.local

The DVC remote is configured as a Google Drive remote (gdrive://...). OAuth credentials are stored locally in .dvc/config.local. Do not commit .dvc/config.local to Git. If Google Drive API access was just enabled, wait a few minutes and retry dvc push -v.

Run the pipeline from WSL if desired, then use PowerShell to sync artifacts:

dvc push -v

DuckDB CLI

DuckDB is used to query .parquet and other table-like artifacts.

Installation

curl https://install.duckdb.org | sh

Run

Open or create a local DuckDB database, create a view over the aggregated Parquet file, and summarize the dataset:

duckdb results/duckdb/h3n2.duckdb
CREATE OR REPLACE VIEW variants AS
   SELECT *
   FROM read_parquet('results/aggregated/aggregated_results.parquet');
h3n2 D SUMMARIZE variants;

Output

┌──────────────────┬─────────────┬───────────────────────┬──────────────────────┬───────────────┬────────────────────┬───┬────────────────────┬─────────────────────┬────────────────────┬───────┬─────────────────┐
│   column_name    │ column_type │          min          │         max          │ approx_unique │        avg         │ … │        q25         │         q50         │        q75         │ count │ null_percentage │
│     varchar      │   varchar   │        varchar        │       varchar        │     int64     │      varchar       │ … │      varchar       │       varchar       │      varchar       │ int64 │  decimal(9,2)   │
├──────────────────┼─────────────┼───────────────────────┼──────────────────────┼───────────────┼────────────────────┼───┼────────────────────┼─────────────────────┼────────────────────┼───────┼─────────────────┤
│ Stability_Score  │ DOUBLE      │ 294.126               │ 388.008              │            23 │ 326.20450000000005 │ … │ 308.959            │ 322.992             │ 337.638            │    38 │            0.00 │
│ Hamming_Distance │ DOUBLE      │ 300.0                 │ 301.0                │             2 │ 300.86842105263156 │ … │ 301.0              │ 301.0               │ 301.0              │    38 │            0.00 │
│ CpG_Ratio        │ DOUBLE      │ 0.0                   │ 0.481403740477819    │            18 │ 0.2792196003407591 │ … │ 0.0                │ 0.45080824050764723 │ 0.4680314143534351 │    38 │            0.00 │
│ ESM2_Perplexity  │ DOUBLE      │ 1.2484898567199707    │ 1.2595747709274292   │            24 │ 1.254385609375803  │ … │ 1.2521368265151978 │ 1.2536463141441345  │ 1.2575433254241943 │    38 │            0.00 │
│ Variant_ID       │ VARCHAR     │ A_Badajoz_18773566_H… │ EPI5184759_HA_A_Fra… │            28 │ NULL               │ … │ NULL               │ NULL                │ NULL               │    38 │            0.00 │
└──────────────────┴─────────────┴───────────────────────┴──────────────────────┴───────────────┴────────────────────┴───┴────────────────────┴─────────────────────┴────────────────────┴───────┴─────────────────┘

---

Copyright (c) 2026 @thiyageshvenkat