SMHI — Demo & Scripts

This repository provides SMHI training and external evaluation code, plus a minimal demo dataset and one-click scripts.

Contents

• SMHI module (smhi/): python package (training + external evaluation)
• bin/: command-line entrypoints (thin wrappers)
• scripts/: runnable shell scripts (relative paths)
• demo_data/: minimal demo CSVs

Method overview (academic description)

This code implements a Top-(K) feature selection and 1D convolutional neural network with batch normalization (BN) for supervised classification on microbiome abundance profiles. Model training is performed using stratified cross-validation, producing per-fold models and aggregated predictions. External cohort evaluation aligns features to the training feature space, reports missing features, and optionally applies probability calibration before generating standard performance visualizations and per-sample outputs.

Inputs and outputs

• Input (training):
  • Abundance table CSV (rows = samples, columns = microbial features; first column is seq.id)
  • Metadata CSV containing disease.type (first column is seq.id; HC is treated as control, all other labels are treated as case)
• Input (external evaluation):
  • External cohort abundance + metadata in the same format
  • Training feature list exported by the training run (model/profile_filtered.csv)
• Outputs:
  • Trained models (per fold), cross-validated predictions, threshold table, and summary plots under outputs/.../<target>/model/
  • External-cohort predictions and reports under outputs/.../<target>/result/<cohort>/

Quickstart (demo)

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

bash scripts/demo_run.sh

Outputs will be written to ./outputs/demo_run/.

Expected demo outputs (file-level checklist)

After running bash scripts/demo_run.sh, the following files are expected (paths relative to repository root):

• Training artifacts (outputs/demo_run/Top100_BN_demo/model/)
  • parameters.json: exact hyperparameters and filtering configuration used for the run
  • profile_filtered.csv: filtered and transformed feature matrix used for training (also serves as the external feature list)
  • optimal_thresholds.csv: per-fold operating thresholds
  • Train_predictions_mean.csv, Val_predictions_mean.csv: mean cross-validated predicted probabilities (and SMHI) per sample
  • CNN_best_model_0.pt ... CNN_best_model_2.pt: per-fold trained model weights (demo uses 3 folds)
  • train_roc.pdf, val_roc.pdf: ROC curves
  • Train_probability_violin.pdf, Val_probability_violin.pdf: probability distribution plots
  • Train_SMHI_violin.pdf, Val_SMHI_violin.pdf: SMHI distribution plots
• External evaluation artifacts (outputs/demo_run/Top100_BN_demo/result/External_demo/)
  • External_demo_missing_features.csv: list of training features absent from the external cohort (filled with zeros)
  • External_demo_feature_matching_report.csv: feature overlap statistics (common/missing/extra)
  • External_demo_External_predictions_mean.csv: per-sample mean external predicted probabilities (and SMHI)
  • External_demo_External_roc.pdf: external ROC curve (if both classes exist)
  • External_demo_External_predictions_mean_calibrated.csv: calibrated external mean predictions (enabled in demo)

Environment (versions tested)

The software stack below corresponds to the Conda environment python3.11 on macOS (osx-64) used for validation.

• Python: 3.11.10
• numpy: 2.1.2
• pandas: 2.2.3
• PyTorch (torch): 2.2.2
• scikit-learn: 1.6.1
• SciPy: 1.14.1
• matplotlib: 3.9.2
• seaborn: 0.13.2

Notes:

• The validated PyTorch build is CPU-only on macOS (osx-64).

Installation time and demo runtime (estimates)

• Install time: ~3 minutes in a fresh .venv using Python 3.11 on macOS (standard broadband; depends on whether PyTorch and NumPy wheels are cached).
• Demo runtime: typically < 5 minutes on a laptop/desktop CPU (the demo uses a small dataset, 3-fold CV, and 10 epochs).

Hardware/software requirements (demo)

• OS: macOS (osx-64) validated; Linux should be supported with the same Python stack.
• RAM: (\ge) 4 GB is sufficient for the demo dataset.
• GPU: not required for the demo.

Using your own data (cross-cohort evaluation)

1. Prepare training input files:
   • train_profile.csv: abundance table with seq.id as sample identifier (first column), features as columns.
   • train_metadata.csv: metadata with seq.id (first column) and a disease.type column.
2. Train the model (example):

python -m smhi.train \
  --prof_path /path/to/train_profile.csv \
  --meta_path /path/to/train_metadata.csv \
  --output_dir /path/to/outputs \
  --target MyTop100BN \
  --top_species 100 \
  --model_type bn_enhanced \
  --use_mask

3. Run external cohort evaluation using the exported training feature list:

python -m smhi.external_eval \
  --prof_path /path/to/external_profile.csv \
  --meta_path /path/to/external_metadata.csv \
  --model_dir /path/to/outputs/MyTop100BN/model \
  --feature_list /path/to/outputs/MyTop100BN/model/profile_filtered.csv \
  --output_dir /path/to/outputs/MyTop100BN/result/ExternalCohort \
  --target ExternalCohort \
  --train_results /path/to/outputs/MyTop100BN/model/Val_predictions_all.csv \
  --calibration_method percentile_mapping \
  --use_mask

Key points:

• External features are aligned to the training feature space; missing training features are recorded in *_missing_features.csv and filled with zeros.
• If the training run used --use_mask, external evaluation should also pass --use_mask to ensure input dimensionality matches.