Many heuristic / randomized solvers (simulated annealing, beam search, genetic, etc.) have meaningful run-to-run variance — from RNG seed and from wall-clock-bounded iteration counts. A single evaluation per config is therefore an unreliable signal, often ±0.5–1%.
Today experiment optimize (Optuna) and ad-hoc config comparison treat each trial's metric as one reliable scalar. With a noisy metric this has two failure modes:
- Optimization chases noise — Optuna picks the config that drew a lucky run, not the genuinely-better one.
- A/B comparisons mislead — comparing two configs on one run each produces "wins" that flip when re-run.
Proposed features (any subset is useful):
- Multi-run averaging per evaluation — run each config/trial N times, aggregate (mean/median + spread), so the optimizer sees a stable metric.
- Paired A/B comparison — run two configs interleaved over the test set R times; report the per-round paired diff and sign-consistency (how many rounds A beats B), not just the mean — sign-consistency is far more robust than the mean for small effects.
- (optional) A-vs-A noise-floor calibration — run a config against itself to measure inherent variance, so the user knows which effect sizes are detectable vs. noise.
This makes hyperparameter tuning and config comparison trustworthy for the randomized solvers common in heuristic-optimization work, and avoids the "tuned on a single lucky run" trap. Composes naturally with a multi-case suite runner (separate request).
Many heuristic / randomized solvers (simulated annealing, beam search, genetic, etc.) have meaningful run-to-run variance — from RNG seed and from wall-clock-bounded iteration counts. A single evaluation per config is therefore an unreliable signal, often ±0.5–1%.
Today
experiment optimize(Optuna) and ad-hoc config comparison treat each trial's metric as one reliable scalar. With a noisy metric this has two failure modes:Proposed features (any subset is useful):
This makes hyperparameter tuning and config comparison trustworthy for the randomized solvers common in heuristic-optimization work, and avoids the "tuned on a single lucky run" trap. Composes naturally with a multi-case suite runner (separate request).