mert_cft

This repository is a minimal handoff package centered on the successful MERT+CFT experiment:

• run id: 20260614_011400_COnP
• training script: train_conp_v6_0415.py
• config: config_mert_base.yaml
• model selection target: COnP F1

It intentionally excludes unrelated experiments, alternative batch-size runs, historical checkpoints, and bulky output folders.

Included

• train_conp_v6_0415.py: training entry
• model.py: MERT frontend + CFT model
• dataset.py: MIR-ST500 dataset loader with 3-source mixed training support
• predict_to_json.py: prediction export
• evaluate_github.py: evaluation script
• config_mert_base.yaml: exact config used by the target run
• data/MIR-ST500_corrected.json: label file used by this setup
• splits_v11/: split files used by this run
• splits_solea/: SOLEA split files (train only; val/test empty — SOLEA only contributes training, never eval)
• splits_solea_smoke/: small SOLEA subset for smoke test
• run_records/20260614_011400_COnP/: key logs from the target experiment

Exact Training Conditions

Evidence comes from run_records/20260614_011400_COnP/logs/train_stdout.log.

• Dataset input: waveform
• Sample rate: 24000
• Segment frames: 512
• Infer chunk frames: 512
• Max samples per epoch: 12000 (baseline) → 20000 (SOLEA run)
• Train mix sources: original + pitch_shift + pitch_shift_complement (+ solea in the SOLEA run)
• Frontend: MERT-v1-95M
• Frontend freeze feature encoder: true
• Batch size: 8
• Num workers: 4 (baseline) → 24 (SOLEA run)
• Learning rate: 1e-4
• Backbone learning rate: 1e-5
• Epochs: 1300
• Scheduler: CosineAnnealingLR
• AMP: enabled
• Best-model criterion: COnP F1

Main Result Snapshot

From run_records/20260614_011400_COnP/test_monitor.txt:

• best test-monitor COnP_f1: 0.810985 at epoch 8
• best test-monitor COnPOff_f1: 0.638390 at epoch 12
• best test-monitor COn_f1: 0.837046 at epoch 8

The validation-side log continued improving later, but the monitored holdout test peak for COnP in the retained record is epoch 8.

Paths You Must Adjust

config_mert_base.yaml still points to the original local data/model paths. Before running on a new machine, update at least:

• data.audio_dir
• data.label_path
• data.splits_dir
• data.cqt_cache_dir
• data.train_mix_sources[*]
• model.wav2vec_path
• training.run_dir

Run

Use the same environment style recorded in:

• run_records/20260614_011400_COnP/reproduce_successful_run.md

Training command:

python3 train_conp_v6_0415.py --config config_mert_base.yaml

Notes

• This package is intentionally narrow: it is for this MERT+CFT experiment only.
• No other experiments were copied into this repository.

---

Update 2026-07-07 — SOLEA data integration (offset-masked training)

What changed (vs. baseline 315f440)

This update integrates a new in-domain singing dataset, SOLEA (flamenco vocal excerpts, datasets/17548vocal/, ~7.7k training segments), into the mixed-source training pipeline. SOLEA only contributes onset / frame supervision; its offset annotations are not trustworthy (uncorrected score alignment), so they are masked out of the offset loss.

1. dataset.py — multi-source loader + offset-trust mask

• New per-source field offset_trust (default true). SOLEA is configured false.
• __getitem__ now returns a 3-tuple (input, labels, offset_mask) where offset_mask is a per-sample bool (true = offset supervision trusted).
• Short-segment safety: segments whose label frames < segment_frames (SOLEA has many clips shorter than 12.8 s) are zero-padded to segment_frames so batch torch.stack no longer fails on unequal frame counts.
• Audio path resolution now also tries *_vocals.flac / *.flac (SOLEA ships FLAC).
• Waveform-mode source scan uses a ThreadPoolExecutor (≤16 threads) for parallel soundfile.info probing, removing a serial startup bottleneck.

2. model.py — masked offset loss in CFTLoss

• forward accepts an optional offset_mask.
• With mask (training): offset loss is the per-sample mean weighted by the trust mask, i.e. SOLEA samples contribute zero offset gradient but keep a graph connection (avoids AMP "unused parameter" errors). All-trusted batches behave exactly as before.
• Without mask (validation / threshold search): unchanged, plain .mean().

3. train_conp_v6_0415.py

• Train loop unpacks the new offset_mask and passes it to the criterion.
• Test-monitor trigger rule rewritten:
  • epochs 1–12: monitor at epochs 1, 4, 7, 10 (every 3, regardless of best) → early visibility
  • epochs ≥ 13: monitor only on a new validation-best (the old every-40 periodic trigger removed)

4. Configs

• config_mert_base.yaml: adds solea source (offset_trust: false), max_samples_per_epoch 12000 → 20000, num_workers 4 → 24, threshold_workers 12 → 64.
• config_mert_smoke.yaml: adds a train_mix_sources block (original + solea-smoke) so the 20-min smoke job actually exercises the new code path.

5. Slurm & splits

• run_25ms_smoke20_gpuh08.slurm: cpus 16 → 80, runs config_mert_smoke.yaml.
• run_25ms_ta_2h_gpuh08.slurm: cpus 16 → 80, node gpuh08 → gpuh09.
• New splits_solea/ (7709 train, empty val/test) and splits_solea_smoke/ (50 train).
• Added the reference paper REFINING AUDIO-TO-SCORE ALIGNMENT FOR SINGING VOICE.pdf.

Experiment status

SOLEA-integrated 2 h run on gpuh09 (slurm job 90014, config_mert_base.yaml, 4 sources including SOLEA, 57745 train samples, max_batches 2500/epoch). Training was healthy and improving when the wall-clock limit hit at epoch 22 — cancelled due to TIME LIMIT, not a code failure. Validation metrics progressed steadily:

epoch	val COn_f1	val COnP_f1	val COnPOff_f1	test-monitor COnP_f1
1	0.8003	0.7709	0.5462	0.7707
8	0.8518	0.8379	0.6739	0.8197 (ep13)
16	0.8781	0.8699	0.7299	—
22	0.8914+	0.8848+	0.7446+	~0.815–0.820

Reference baseline (3-source, no SOLEA, job 89949) at the same wall-clock reached ~COnP_f1 0.8424 at epoch 9 / 0.8379 at epoch 8 — SOLEA-augmented training is on a faster and higher trajectory on validation, with offset metrics also rising despite SOLEA being offset-masked (the trusted sources still drive the offset head).

Test-monitor COnP on the holdout (MIR-ST100) hovered ~0.81–0.82, similar to baseline at this early stage; the run needs more epochs (it is configured for 1300) before the SOLEA benefit shows on the held-out test set.

Known issue (fixed in this update)

An earlier SOLEA run (90005) crashed in the DataLoader with stack expects each tensor to be equal size, got [395,48] vs [512,48] — caused by short SOLEA clips producing sub-segment_frames labels. The zero-pad fix in dataset.py.__getitem__ resolves this; 90014 ran clean afterwards.

Reproduce

# full run
python3 train_conp_v6_0415.py --config config_mert_base.yaml
# smoke (20 min, exercises solea code path)
python3 train_conp_v6_0415.py --config config_mert_smoke.yaml