In the current codebase, the validation set is used both for early stopping and for reporting / comparing final results.
This introduces validation leakage:
the model is effectively tuned to the validation set twice — first by selecting the stopping epoch, and then by selecting or comparing models based on the same validation metrics.
As a result, the reported validation performance is optimistically biased and likely higher than what would be observed on truly unseen data.
Why this is a problem:
Early stopping is a form of model selection.
Any dataset used for model selection should not be reused for final performance estimation.
This leads to overestimated results and reduced reproducibility.
Suggested fixes:
Use a separate test set for final evaluation, keeping validation strictly for early stopping and hyperparameter tuning.
Alternatively, use nested cross-validation, where early stopping is done in inner folds and performance is reported on outer folds.
If only one split is available, clearly state that validation results are not unbiased estimates of generalization.
Clarifying or fixing this evaluation protocol would make the reported results more reliable and comparable.
In the current codebase, the validation set is used both for early stopping and for reporting / comparing final results.
This introduces validation leakage:
the model is effectively tuned to the validation set twice — first by selecting the stopping epoch, and then by selecting or comparing models based on the same validation metrics.
As a result, the reported validation performance is optimistically biased and likely higher than what would be observed on truly unseen data.
Why this is a problem:
Early stopping is a form of model selection.
Any dataset used for model selection should not be reused for final performance estimation.
This leads to overestimated results and reduced reproducibility.
Suggested fixes:
Use a separate test set for final evaluation, keeping validation strictly for early stopping and hyperparameter tuning.
Alternatively, use nested cross-validation, where early stopping is done in inner folds and performance is reported on outer folds.
If only one split is available, clearly state that validation results are not unbiased estimates of generalization.
Clarifying or fixing this evaluation protocol would make the reported results more reliable and comparable.