DNNGP_Pro

Multi-omics data integration platform for crop breeding from complex trait modeling through variety development

We constructed the DNNGP_Pro platform for predictive modeling of complex hybrid traits in large-scale populations. In preliminary steps, the pipeline extracts local features from omics data via convolutional neural networks, then employs graph convolutional networks to integrate these single-omics features with a parental relationship-based graph, and captures long-range dependencies across multi-omics features using a Transformer module. Exhibiting comparable performance to DNNGP model on single-omics data.

DNNGP_Pro provides:

• Python API (DNNGPProRegressor) — training, prediction, evaluation, explainability, and Optuna hyperparameter search (see the Notebook for usage examples)
• Command-line tools — cli_train.py, test.py for batch experiments and HPC; hyperparameter search via API optimize()
• Reproducible CV — default 5-fold × 3-repeat cross-validation; checkpoints store y_scaler and model metadata

---

Related Software and Tools

• DNNGP – Deep neural network for genomic prediction.
• EXGEP – Predicting genotype-by-environment interactions with explainable ML ensembles.
• GxEtoolkit – Automated explainable ML for genomic prediction.
• AutoGS – Automated genomic selection workflows.

---

Operation systems

• Windows
• Linux

---

Table of Contents

• Getting started
• Data format
• Basic API usage
• Command-line usage
• Hyperparameter search
• Explainability
• Hyperparameter optimization
• Outputs
• Project layout
• Citation
• Copyright and License
• Contacts

---

Getting started

Requirements

• Python 3.9+ (3.9.23 tested)
• PyTorch (install a build matching your CUDA version)
• NumPy, pandas, scikit-learn, Optuna (for HPO)

Installation

1. Create a Python environment.

conda create -n dnngp_pro python=3.9
conda activate dnngp_pro

2. Clone this repository and install dependencies.

git clone https://github.com/AIBreeding/DNNGP_Pro.git
cd DNNGP_Pro
pip install -r requirements.txt

3. For API/scripts, add DNNGP_Pro to the path (or run from the project root as in Notebook below).

---

Data format

All tables are tab-separated (.txt) unless you pass sep= in the API.

Role	Column / rule
Sample ID	ID on every omics and phenotype file
Phenotype	One numeric column besides ID (e.g. kgwmean in kgwmean.txt)
Pedigree (optional)	P1_name, P2_name on an omics file (often genome)

Default directory layout (DNNGP_Pro/config.py):

dataset/training/
├── Genomescale.txt
├── Metabolomescale.txt
├── Transcriptomescale.txt
└── kgwmean.txt

dataset/testing/ # independent test (There is no requirement for identical filenames)
├── Genomescale.txt
├── Metabolomescale.txt
├── Transcriptomescale.txt
└── kgwmean.txt

Omics keys (API / CLI --omics):

Key	Typical file
genome	Genomescale.txt
metabo	Metabolomescale.txt
transc	Transcriptomescale.txt

Use one, two, or three omics (e.g. --omics genome,metabo or only genome in omics_paths).

Important for independent prediction: training and test omics must use the same feature columns and order (same number of numeric columns per modality). Mismatched dimensions cause checkpoint load errors.

---

CLI python cli_train.py --run_baselines additionally trains per-omics CNN, multi-omics CNN-only, per-omics CNN–GCN, and the main Fusion model (when parent is available).

API fit() trains one primary model per call (fusion or CNN-only), matching parent_source and omics count.

---

Basic API usage

Full workflows (multi/single omics, with/without parent, HPO, IG) are in Notebook.

Train

import pandas as pd
from DNNGP_Pro.adapters import from_paths
from DNNGP_Pro import DNNGPProRegressor

GMT_model = DNNGPProRegressor(
    n_folds=5,
    n_repeats=3,
    n_epochs=300,
    fusion_mode="concat",
    output_dir="outputs",
    device="auto",
    seed=42,
)

GMT_model.fit(
    omics_paths={
        "genome": "dataset/training/Genomescale.txt",
        "metabo": "dataset/training/Metabolomescale.txt",
        "transc": "dataset/training/Transcriptomescale.txt",
    },
    phenotype_path="dataset/training/kgwmean.txt",
    parent_source=None,   # use "genome" for parent graph  + Fusion
    sep="\t",
)

GMT_model.save("outputs/state.json")

Test

inp = from_paths(
    omics_paths={
        "genome": "dataset/testing/Genomescale.txt",
        "metabo": "dataset/testing/Metabolomescale.txt",
        "transc": "dataset/testing/Transcriptomescale.txt",
    },
    parent_source=None,
    sep="\t",
    # phenotype_path optional — IDs from omics intersection only
)

pred = GMT_model.predict(
    data=inp,
    save_path="outputs/test_predictions.csv",
)

# Evaluate only when you have ground truth (separate step)
truth = pd.read_csv("dataset/testing/kgwmean.txt", sep="\t").set_index("ID").loc[inp.sample_ids].iloc[:, 0].values
metrics = GMT_model.score(pred, truth)
print(metrics)

---

Command-line usage

Training — cli_train.py

Fixed hyperparameters, optional full baseline suite, optional refit on all data and ensemble checkpoints.

python cli_train.py \
  --data_dir dataset/training/ \
  --output_dir outputs \
  --omics genome,metabo,transc \
  --fusion_mode concat \
  --graph_edge_mode any \
  --run_baselines \
  --device auto \
  --seed 42

Other fusion modes: gated_residual, bilinear, cross_attn.
Without P1_name/P2_name in data, Fusion/GCN blocks are skipped; use --run_baselines for CNN models.

Results: outputs/results.json, per-model folders under outputs/, best_model.pt per run.

Independent test — test.py

Loads train + test directories, builds a transductive kinship graph on train ∪ test when parents exist, evaluates all model subfolders under --model_dir.

python test.py \
  --train_dir dataset/training/ \
  --test_dir dataset/testing/ \
  --model_dir outputs \
  --output_dir test_outputs \
  --omics genome,metabo,transc \
  --device auto

Options: --prefer_full, --use_ensemble, --fusion_dir.
Note: test.py expects test phenotypes for metrics; for label-free API workflow use predict + score(pred, y_true) as above.

Hyperparameter search (API only)

Use DNNGPProRegressor.optimize() (Optuna). Each trial runs CV under outputs/optuna_trials/trial_XXXX/. Artifacts: save_path/trials.csv, best_params.json.

GMT_model.optimize(
    omics_paths={...},
    phenotype_path="dataset/training/kgwmean.txt",
    parent_source="genome",   # or None for CNN-only search
    n_trials=20,
    save_path="optuna_out",
)

Each trial runs **n_folds × n_repeats** CV fits (default 15). Trial checkpoints: `outputs/optuna_trials/trial_XXXX/`.

After search, refit with best hyperparameters (set_params + fit),see Notebook.

---

Explainability

Via API only (model.explain); see Notebook.

level	method	Meaning
omics	proxy_signal / integrated_gradients	Modality-level importance
feature	proxy_signal / integrated_gradients	Per-feature IG (mean over samples); outputs per omics + feature_ig_explanation_all.csv
sample	integrated_gradients	Per-sample omics contribution
sample_feature	integrated_gradients	Per-sample × per-feature IG (slow)
internal	gate	Fusion / GCN gate weights from checkpoint

GMT_model.explain(
    data=inp,
    level="feature",
    method="integrated_gradients",
    top_k=50,
    steps=16,
    save_path="explain/feature_ig",
)

proxy_signal is input magnitude only — not model attribution. Use IG in publications.

---

Outputs

Path	Description
outputs/{ModelName}/best_model.pt	CV-best weights + y_scaler, fusion_config, model_family
outputs/genome-CNN/	Single-omics CNN (no pedigree)
outputs/CNN-only_genome+metabo+transc/	Multi-omics CNN-only
outputs/CNN-GCN-Fusion_*	Fusion with pedigree
outputs/state.json	API metadata (ckpt_path_, params, train_dir_)
outputs/results.json	CLI summary
test_outputs/*_predictions.csv	CLI test.py predictions

---

Project layout

DNNGP_Pro/
├── cli_train.py          # Training + optional baselines
├── test.py               # Independent evaluation (CLI)
├── Notebook/              # Usage examples
├── dataset/              # Example training data
└── DNNGP_Pro/
    ├── api.py            # DNNGPProRegressor
    ├── adapters.py       # from_paths, directory loading
    ├── explain.py
    ├── optimize.py
    ├── training.py
    ├── inference.py
    ├── models.py
    ├── data.py
    ├── types.py
    ├── training.py
    ├── utils.py
    └── config.py

---

Citation

You can read our paper explaining DNNGP_Pro (xxx).

When reporting results, please specify: omics combination, sample count after ID intersection, use of pedigree (parent_source, graph_edge_mode), CV settings (n_folds, n_repeats), test protocol, and attribution method (IG vs proxy).

---

Copyright and License

This project is licensed under the GNU General Public License v3.0 (GPLv3) — see the LICENSE file for details.

---

Contacts

For more information, please contact Huihui Li (lihuihui@caas.cn).

---

Train multi-omics and single-omics models, integrated parental graphs, hyperparameter optimization, and model interpretation

Refer to Notebook for complete examples.