Tian's dataset seems not converging.

[biocomp4dev]

Hey team, thank you for your awesome tool and tutorial!

I'm probably missing something in my setup, but the Tian dataset (single cell type, used in Figure 4 of original paper) isn't converging for me. Random 80/20 splits give test AUC ~0.5 (random level). I have also tried more steps than tutorial (3000 steps), but the same issue appeared. I have conformed that this configuration works on the example dataset provided by the tutorial. Any tips on common config pitfalls?

My steps:

1. Load Tian CRISPRi data (or CRISPRa).
2. Split 80/20 randomly
3. Train default model 3000 steps.
4. Evaluate on the test dataset, test AUC: consistently ~0.5 across runs.

My config:

{
    "global_batch_size": 128,
    "local_batch_size": 4,
    "mixed_precision": "true",
    "nr_step": 3000,
    "warmup_step": 500,
    "lr": 0.001,
    "chk_time_interval": 3600,
    "chk_step_interval": 200,
    
    "dataset_dir": $dataset_dir, ## directory of the training and testing dataset
    "log_dir": $log, ## directory of the training log
    "model_dir": $model, ## directory of the fine-tuing checkpoints
    "pretrain_model_dir": $pretrain, ## directory of pretrain_weights.pth

    "train_data": $train_data,
    "test_data": $test_data,
    "dist_graph" : "dist_t_matrix.csv",
    "adj_graph" : "adj_t_matrix.csv"
}

Env: PyTorch 2.4.0, single RTX 4090D (24G Mem), Ubuntu 22.04.

[wcanisay-y]

Thanks a lot for sharing your config and setup. We have the following comments that may help you run our tool successfully.

1. Please ensure that the Tian dataset is preprocessed exactly according to the steps described in the paper. If possible, double‑check that the final training labels are not severely imbalanced between classes in your split. Note that carving out a separate validation set may slightly reduce the performance measured on the test set.

2. The dist_t_matrix.csv and adj_t_matrix.csv files currently in the repo are only meant for the example dataset and should not be reused directly for the Tian data. Instead, for each real dataset (including Tian), these two files need to be regenerated to reflect the correct sample graph for that dataset. Otherwise, the model effectively sees the wrong graph structure and tends to converge to random performance. The detailed steps for generating dist_t_matrix.csv and adj_t_matrix.csv are in issue #1.

3. For larger real datasets like Tian, the default hyperparameters tuned on the tiny example set are usually not the best. We recommend increase warmup_step, slightly decrease the initial lr, and train for more total steps before evaluating, usually up to 30,000.

Hope that helps!

[biocomp4dev]

Thank you for the helpful suggestions! I am still facing challenge to fix this issue.

1. The Tian dataset from scPerturb Zenodo was filtered using the Fernandes preprocessing script and further removing conditions with <50 cells, aligning with CellNavi paper methods. Training/test splits show balanced perturbed cell distributions across conditions (see attached figures). Could you confirm if these class balances look suitable, or suggest validation set adjustments?

2. I noticed that the provided GRN contains 52% (11,291/21,829) genes of Tian's dataset. Comparing with the 62% (11,936/19,240) coverage of provided example Schimidt's dataset of T cell. It seems wired that a feature drop of only 645 genes causes collapse in performance since the GRN prior from NicheNet is only cropped/filtered for the new dataset as mentioned in Question about “single-cell object” vs. prior graph (NicheNet) #3 and the distance matrices were computed once on the globally integrated graph as mentioned in Pretrained adjacency matrix and distance matrices for the prior gene graph #5, unless almost informative genes are in the uncovered genes. Considering the dataset-specific GRN is important to obtain optimal results and many users enquire the construction of it, could you please provide a tutorial for this step?

3. In my first several trial, CellNavi tooks more than 3.5 h for training on 23k cells and about 20min for evaluation on only 5k cells using above configuration, so it would take more than 35h for training with 30,000 steps. However, Tian dataset is relatively small comparing to Schmidt dataset (50k cells), Norman dataset (101k cells), and Replogle dataset (494k cells), so up to 30,000 steps for large datasets will consume significant long time to obtain a reasonable applicability. May you provide recommendation to speedup the training and evaluation process with limited computational resources (4090D 24G memory)?

Best regards