Skip to content

Stabilize OptimizationBBO multi-objective tests#1277

Merged
ChrisRackauckas merged 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:codex/investigate-bbo-objective-flake
Jul 20, 2026
Merged

Stabilize OptimizationBBO multi-objective tests#1277
ChrisRackauckas merged 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:codex/investigate-bbo-objective-flake

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member

Ignore this PR until reviewed by @ChrisRackauckas.

Summary

  • Seed Julia's RNG immediately before each OptimizationBBO multi-objective solve.
  • Keep the existing expected objective values and tolerances unchanged.
  • Make the random initial population independent of an earlier time-limited solve and machine load.

Investigation

The current downgrade workflow reached the OptimizationBBO tests and exposed a pre-existing stochastic failure. The trigger commit changed only the root Optim compat; its diff contains no OptimizationBBO, OptimizationBase, or downgrade-workflow changes.

History analysis identifies a8299ec5b (2024-08-18) as the commit that added the exact multi-objective assertions after the pre-existing maxtime = 5 solve. That timed solve advances global RNG state by a load-dependent amount. BlackBoxOptim constructs its randomized initial population before applying its run-time seed, so the later asserted objectives varied across clean processes and CI machines. A 100-seed reproduction produced an assertion failure at seed 668073.

The fix seeds immediately before each solve, covering population construction as well as the subsequent optimizer run. It does not change production behavior, test tolerances, expected values, or test counts.

Local verification

  • Recreated the exact downgrade graph with Julia 1.10.11: BlackBoxOptim 0.6.3, OptimizationBase 5.2.1, SciMLBase 2.122.1.
  • Ran the CI-equivalent OptimizationBBO Core command twice independently: 25 passed / 25 total both times, with byte-identical Rosenbrock/Ackley objectives.
  • Ran the same Core tests with the failed job's SciMLTesting 2.2.0 harness: 25 passed / 25 total.
  • Ran the current graph on Julia 1.12.6 with BlackBoxOptim 0.6.9 and SciMLBase 3.36.0: 25 passed / 25 total.
  • Ran repository-wide Runic: exit 0.
  • Ran git diff --check: clean.

Process notes

  1. Reproduced the exact clean-master command rather than attributing the failure to the compat-only trigger commit.
  2. Sampled independent seeds to establish stochasticity without weakening the assertion.
  3. Traced the RNG ordering in BlackBoxOptim and the test's history.
  4. Applied the smallest deterministic test setup and reran exact, pinned-harness, current, and formatting checks.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>

Copy link
Copy Markdown
Member Author

Scratchpad for commit a38b14f2c:

  • Confirmed the triggering master commit has no diff under lib/OptimizationBBO, lib/OptimizationBase, or the downgrade workflow.
  • Equivalent history bisect: a8299ec5b added the exact MOO assertions after an already-present maxtime = 5 solve.
  • Repeated clean processes and a 100-seed sample established that the asserted objectives depended on load-sensitive global RNG state; seed 668073 reproduced an assertion miss.
  • The commit only inserts Random.seed!(1234) immediately before the four MOO solves. No tolerance, expected value, production code, or test-count changes.
  • Local terminal results: exact downgrade Core 25/25 twice, pinned SciMLTesting 2.2 Core 25/25, current Julia 1.12.6 Core 25/25, repository-wide Runic exit 0, and clean diff check.

@ChrisRackauckas
ChrisRackauckas marked this pull request as ready for review July 20, 2026 15:19
@ChrisRackauckas
ChrisRackauckas merged commit 4276a48 into SciML:master Jul 20, 2026
49 of 51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants