Exp/msg scaffold split - #34
Draft
rukubrakov wants to merge 48 commits into
Draft
Conversation
…t index format df_smiles["indexes"] must store local positions in spectrum_indexes, not global MGF indices; filter spectra using the same janssen validation SIMBA applies at load time. Also wire edit_distance.enabled properly into SimilarityModelMultitask via constructor instead of accessing self.config which doesn't exist on that class.
…tion When some spectra pass our preprocessing filter but fail the SIMBA loader's filter at training time, df_smiles rows get dropped. Reset df_smiles index after drop and remap pair_distances mol indices to match the new sequential index, discarding pairs that reference dropped molecules.
When edit_distance is disabled, all ED values are 0, making the existing ED-based sampler degenerate to uniform sampling across 159M heavily skewed pairs. Switch to inverse-frequency binning by MCES similarity so the model sees equal exposure to similar and dissimilar pairs. Also log val_mces_mae (raw MCES units) in validation to make actual prediction quality visible.
…_pred for all test pairs
…hen test spectra are missing
New preprocessing script splits MSG official train 90/10 by Murcko scaffold, uses MSG official test as-is, and discards MSG official val. _apply_mol_remap removed from training and inference — replaced with a hard error since preprocessing now guarantees all spectrum_indexes are valid. ValMetricsCallback replaces scatter with per-step hexbin (mces_hexbin_step*.png, never overwritten) and logs val_mces_spearman after each validation run.
Preprocessing now includes MSG official val fold as val_official (was discarded). Training loads both val sets and passes them as separate Lightning dataloaders; ValMetricsCallback produces per-set hexbin plots (mces_hexbin_<val_name>_step*.png) and logs val_mces_spearman/scaffold and val_mces_spearman/official separately. LossCallback draws both val curves on the same loss plot. New training script v2 has no early stopping and disables metadata features and learnable multitask weights per Juan Sebastian's feedback.
When some spectra pass preprocessing's filter but fail the training loader's validity check (e.g. centroid check on raw vs matchms-normalized intensities), drop affected molecules and their pairs with a warning instead of crashing. mol_idx_remap is stored on molecule_pairs and applied in prepare_data to filter and remap pair_distances after molecules are dropped.
Add dataloader_idx=0 to SimilarityModelMultitask.validation_step so Lightning doesn't crash when two val dataloaders are provided. Recompute MCES bin indices from raw MCES values (0-20) with width-4 bins [0-4, 4-8, 8-12, 12-16, 16-20] instead of similarity * n_bins which left bins 0-1 always empty.
training_step now returns MCES predictions so ValMetricsCallback can accumulate a rolling train buffer and compute Spearman + MSE alongside both val sets. Hexbin plots now show linear and log scale side by side.
Convert similarity (1 - MCES/40) back to raw MCES before plotting so axes read 0 (identical) to 20 (dissimilar) instead of reversed similarity.
Standalone inference script and SLURM job to run the step-61k checkpoint over all scaffold val, official val, and official test pairs, saving CSVs and balanced 6-panel hexbin plots (GT-MCES-balanced, MAE metric).
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Remove superseded training/inference/preprocessing scripts, keeping only the scaffold split training job, val hexbin inference, and official splits inference. Stage scaffold split prep script and inference SLURM updates.
Restore SLURM scripts that existed on dev (were incorrectly deleted). Rename train_msg_scaffold_split_v2 and val_hexbin_v3 to drop version suffixes. Add experiments/msg_scaffold_split/README.md with motivation, change summary, and step-by-step run instructions.
Pairs with MCES up to 40 are now retained and clipped (similarity = 1 - mces/40), giving the model full coverage of dissimilar pairs. Output dir and downstream SLURM scripts updated accordingly.
Fine resolution in [0,10] (width 2.5) and width-5 bins above, covering the full [0,40] range. Adds n_classes=11 override to training SLURM.
n_classes was hardcoded to 6 inside __init__, overriding the argument. Now respects whatever is passed in.
Was called with no args so always built a 6×6 penalty matrix regardless of config.
…p-29k on asimov2 Inference now runs on the mces40 step-29k checkpoint and generates two sets of 6 plots: all pairs and GT MCES ≤ 20 only.
Override mgf_path from mapping.pkl to use /mnt/data instead of /mnt/data2. All SLURM scripts for asimov now use local disk paths throughout.
ED-based sampling now used even when ED head is disabled, as long as real ED values are present in the preprocessing (detected via non-zero ED column).
Preprocessing script uses scaffold 10% val split and MCES threshold=40; SLURM script mirrors clip-40 settings on new data.
Contributor
|
Caution Review failedAn error occurred during the review process. Please try again later. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ove threshold Mirrors reference clip-40 behaviour: all pairs kept, lb values clipped to MCES_CAP=40.
…nce.py inference.py now remaps /mnt/data2/ -> /mnt/data/ like training.py does, so inference runs cleanly on asimov. Adds four val_hexbin slurm scripts at the best scaffold-val checkpoint for each experiment.
…ces-only Corrected 'SpectraVerse v2' to 'MSG HDF5 (scaffold v2 split)'. Reran metadata inference at step 22k (official val peaks there) and sv2-mces-only at step 67k to match sv2-both. Added dashboard assets.
New distance source takes element-wise max of Gaetan lb_matrix (tight lower bound for MCES >= 10) and HDF5 (exact for MCES < 10), giving best of both. Training capped at 70k steps, all I/O on /mnt/data (asimov local).
Same distance source as prepare_msg_max_lb_hdf5.py but filters out pairs with Morgan fingerprint Tanimoto < 0.2 or either molecule > 40 heavy atoms, matching the original SIMBA fast_compute.py pre-filter exactly.
CE fix (loaders.py): read collision_energy key with safe float parsing; CE/100 normalization in encoder; MCES sampling bug fix in training.py (normalized ED check). Add simba_retrieval.py with --force_ce_zero diagnostic flag. Add compute_mces_exact_1020.py + SLURM array job for exact MCES on 62.6M unresolved [10,20] pairs. Add retrieval results TSVs and dashboard updates.
…board updates apply_exact_mces_1020.py replaces lower-bound distances with exact MCES for all four splits; sanity-checks ≥99% of exact ≥ lb before writing. Early stopping in training.py now monitors scaffold val Spearman (max) instead of validation loss when scaffold val is available. New training SLURM script (no metadata, patience=15) submitted as job 8011.
Experiment wired inline in the research log with loss + metrics plots. Removed from the experiments panel. Renamed 📜 Background → 📖 Research log.
…, MCES diagnostic Fix val/val_official samplers to use per-split inverse-frequency bin weights instead of train's; add oracle retrieval results, MCES distribution plot, research log entries, and SLURM scripts for inference, retrieval, and two new training runs (own-val-weights, lr3e5-bs2048). Add standalone MCES distribution diagnostic script.
…=2048 run Fixes val_scaffold loading val_official npy files (find_file prefix-suffix bug). Adds per-epoch MCES distribution logging for all splits and a bs=2048 training script.
…ebug counts Val/val_official samplers now use fixed seed=0 for reproducible selection each epoch. Hexbin gridsize updated to 16 (2.5-unit bins), log scale removed. MCES histogram bins now match weight computation (searchsorted side=left). limit_val_batches=100. Debug bin-count logs removed from training.py.
…est-test and test-train hexbin scripts
…scripts, and result plots
…in hexbins, drop unused panels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.