Flow generator saves its WORST epoch as the 'best' checkpoint (key_metric='SSIM' with direction inferred from the name) #182

Summary

The flow generator's ModelCheckpoint is configured to maximize a metric that is actually loss-like, so the checkpoint saved as "best" is the worst epoch of the run. Since projects.get_weightfile_from_rundir() prefers the "best" checkpoint, flow_generator.weights=latest then hands that worst flow generator to feature-extractor training.

Mechanism

deepethogram/base.py:359-361 infers the optimization direction from the metric's name:

key_metric = lightning_module.metrics.key_metric
mode = "min" if "loss" in key_metric else "max"
monitor = f"val/{key_metric}"

deepethogram/flow_generator/train.py:373 sets:

key_metric = "SSIM"

"SSIM" contains no "loss" substring, so mode="max". But the SSIM written into opticalflow_metrics.h5 is a loss-like quantity where lower is better — it correlates positively with val/loss.

Evidence

From a 21-epoch flow generator run (TinyMotionNet, preset=deg_f):

• val/loss, val/L1, val/SSIM and val/smoothness all have argmin = epoch 17 and argmax = epoch 2
• corr(val/SSIM, val/loss) = +0.64 (they move together, confirming lower-is-better)
• lightning_checkpoints/ contains exactly epoch=2-step=3000.ckpt and last.ckpt

The checkpoint labelled "best" has a 13.6% worse validation loss than the final epoch, and is the worst epoch of the entire run on every reconstruction metric.

Impact

projects.get_weightfile_from_rundir() returns best in preference to last and checkpoint.pt. So the documented path

flow_generator.weights=latest

silently selects the worst flow generator produced by training, and every downstream feature extractor is initialized from it.

Note the classification stages are unaffected: their key_metric is f1_class_mean_nobg, which should indeed be maximized, so the name-based heuristic happens to give the right answer there.

Suggested fix

Make the direction an explicit property of the metric rather than inferring it from a substring of its name — e.g. have the Metrics class carry a key_metric_mode ("min"/"max") that each subclass sets, and use that in base.py. The current heuristic fails silently for any minimized metric not named *loss*.

Version

master @ 1541eec (v0.4.0)