Hi @bowen-gao, first of all, thank you for sharing your work. The ProFSA model is well-implemented, and I successfully replicated the zero-shot benchmarks from the provided release.
I am writing to seek your advice on the fine-tuning reproduction. Despite strictly following the procedures in Appendix E.3, I consistently observe a performance drop compared to the zero-shot baseline. Below is a summary of my reproduction efforts and the specific challenges I've encountered.
What's already reproduced
Zero-shot ProFSA using profsa_release/last.ckpt from profsa.tar.gz:
| Benchmark |
Our result |
Paper |
| Kahraman AUC |
0.7991 |
0.80 ✓ |
| TOUGH-M1 AUC |
0.8173 |
0.82 ✓ |
So the encoder weights and eval pipelines from the release work correctly.
What we couldn't reproduce
We followed Appendix E.3 in detail. Both published setups produce AUCs
below the zero-shot baseline:
| Setup |
Config |
Kahraman |
TOUGH-M1 |
Paper |
| TOUGH-M1 setup |
lr=1e-5, 20 ep, warmup 0.1 |
0.71 |
0.69 |
0.94 |
| Kahraman setup |
lr=3e-7, 5 ep, warmup 0.2 |
0.74 |
0.80 |
0.85 |
Other details we matched:
- Effective batch = 264 (per-GPU 8 × 3 GPUs × accum 11), close to paper's 64×4=256
- DeeplyTough cosine loss
l = -y·p − (1−y)(1−p)
return_rep=True (512-dim encoder output)- Init from
profsa_pre_no_h_220816.pt, verified identical to the zero-shot
release ckpt's pocket_model.* weights layer-by-layer (193/193 layers, max
diff = 0)
- Data: ~610K labeled pairs from a local DeeplyTough-style LMDB; we matched
it to GroupShuffleSplit(n_splits=5, test_size=0.2, random_state=0) fold=0
on RCSB seqclust (30% identity) at 100% overlap — but only ~95-97% of
the structures DeeplyTough originally reports (~5,800 of ~6,019).
In the TOUGH-M1 setup, the best-val checkpoint lands at exactly the step
warmup ends, and val/loss degrades the moment lr reaches peak 1e-5,
suggesting the lr may be too aggressive at this effective batch.
Could you share
-
TOUGH-M1 train/test split file — the list of PDB IDs in each fold
you used for the Table 2 fine-tune, so we can confirm we're on the same
data partition.
-
Fine-tune training LMDB (or pair lists) for TOUGH-M1 — the released
pocket_matching/ directory contains only evaluation data
(test_ligand.lmdb, p5_pockets.lmdb); no training set is included.
-
Hyperparameters not in E.3 that may close the gap:
- Weight decay
- Gradient clipping value
- Dropout during fine-tune (kept at pretraining 0.1 or modified?)
- Random seed used for the reported numbers
- For the Kahraman setup (lr=3e-7), is the training data still TOUGH-M1
train pairs, or a different set?
I understand that specific hyperparameter choices or data partitioning details are often difficult to document comprehensively in a paper. Any insights or pointers regarding the gaps above would be invaluable to our research. I’m happy to share my full experiment logs or training scripts if that helps clarify the discrepancy. Thanks for your time and help!
Hi @bowen-gao, first of all, thank you for sharing your work. The ProFSA model is well-implemented, and I successfully replicated the zero-shot benchmarks from the provided release.
I am writing to seek your advice on the fine-tuning reproduction. Despite strictly following the procedures in Appendix E.3, I consistently observe a performance drop compared to the zero-shot baseline. Below is a summary of my reproduction efforts and the specific challenges I've encountered.
What's already reproduced
Zero-shot ProFSA using
profsa_release/last.ckptfromprofsa.tar.gz:So the encoder weights and eval pipelines from the release work correctly.
What we couldn't reproduce
We followed Appendix E.3 in detail. Both published setups produce AUCs
below the zero-shot baseline:
Other details we matched:
l = -y·p − (1−y)(1−p)return_rep=True(512-dim encoder output)profsa_pre_no_h_220816.pt, verified identical to the zero-shotrelease ckpt's
pocket_model.*weights layer-by-layer (193/193 layers, maxdiff = 0)
it to
GroupShuffleSplit(n_splits=5, test_size=0.2, random_state=0)fold=0on RCSB seqclust (30% identity) at 100% overlap — but only ~95-97% of
the structures DeeplyTough originally reports (
~5,800of~6,019).In the TOUGH-M1 setup, the best-val checkpoint lands at exactly the step
warmup ends, and val/loss degrades the moment lr reaches peak 1e-5,
suggesting the lr may be too aggressive at this effective batch.
Could you share
TOUGH-M1 train/test split file — the list of PDB IDs in each fold
you used for the Table 2 fine-tune, so we can confirm we're on the same
data partition.
Fine-tune training LMDB (or pair lists) for TOUGH-M1 — the released
pocket_matching/directory contains only evaluation data(
test_ligand.lmdb,p5_pockets.lmdb); no training set is included.Hyperparameters not in E.3 that may close the gap:
train pairs, or a different set?
I understand that specific hyperparameter choices or data partitioning details are often difficult to document comprehensively in a paper. Any insights or pointers regarding the gaps above would be invaluable to our research. I’m happy to share my full experiment logs or training scripts if that helps clarify the discrepancy. Thanks for your time and help!