Skip to content

Fix spectralog benchmark cases - #147

Open
bernalde wants to merge 2 commits into
mainfrom
fix/issue-75-spectralog-benchmarks
Open

Fix spectralog benchmark cases#147
bernalde wants to merge 2 commits into
mainfrom
fix/issue-75-spectralog-benchmarks

Conversation

@bernalde

Copy link
Copy Markdown
Member

Summary

  • Add finite source-scale bounds for spectralog objective-value variables so GDPopt/GLOA gets a bounded relaxation instead of fallback val[j] bounds.
  • Keep the existing GDP model structure and public build_model() API intact.
  • Remove spectralog/gdpopt.enumerate from the reusable PR58 local benchmark campaign and document why: 30 independent disjunctions imply 2^30 realizations, and Pyomo currently materializes the enumeration list before enforcing time_limit.
  • Add focused spectralog tests for the derived bounds and Big-M/Hull transformation paths.

Tests run

  • pixi run pytest tests/test_spectralog.py tests/test_benchmark.py::test_committed_pr58_local_cases_cover_default_matrix -v --tb=short -> 4 passed.
  • pixi run gdplib-benchmark run --instances spectralog --strategies gdpopt.gloa --timelimit 60 --solver-profile gams-local --run-id issue75_gloa_fixed --no-summary --no-skip-existing --fail-fast -> 1 row, 0 failures; GDPopt/GLOA optimal in 2 iterations, LB = UB = 15.540610755468409.
  • pixi run pytest tests/test_module_imports.py -v --tb=short -> 74 passed.
  • pixi run pytest tests/test_benchmark.py -v --tb=short -> 21 passed.
  • pixi run lint-black -> passed.
  • pixi run lint-flake8-critical -> passed, 0 critical errors.
  • pixi run lint-flake8 -> exit 0; existing style inventory remains.
  • pixi run typos --config ./.github/workflows/typos.toml -> passed.
  • pixi run test -> 338 passed, 1 skipped.
  • git diff --check -> passed.

Notes

  • I did not run spectralog with gdpopt.enumerate to completion. A focused reproduction exceeded the requested 60-second solve limit before reaching a solve result, matching the known Pyomo behavior that enumeration materializes all discrete realizations before honoring time_limit.
  • Issue Plan PyPI release workflow and Pixi platform support #104 is closed release/Pixi policy work and this PR leaves packaging, release workflow, and Pixi platform support unchanged.

Closes #75

@bernalde

Copy link
Copy Markdown
Member Author

Benchmark follow-up for spectralog after adding finite val[j] bounds.

I ran the requested 5-minute matrix for this instance: reformulations plus GDPOpt except gdpopt.enumerate, across local, Gurobi, and global solver profiles.

Run id:

issue75_spectralog_5min_matrix_20260514

Result summary:

Strategy Local Gurobi Global
gdp.bigm optimal, obj 12.089262, ~0.4s optimal, obj 12.089262, ~6.7s optimal, obj 12.089262, ~2.3s
gdp.hull optimal, obj 12.089262, ~0.2s optimal, obj 12.089262, ~6.8s optimal, obj 12.089262, ~2.7s
gdpopt.loa maxTimeLimit, UB 59.326029, LB 4.0 maxTimeLimit, UB 59.326029, LB 4.0 maxTimeLimit, UB 12.089262, LB 4.0
gdpopt.gloa optimal, obj 15.540611, ~3.6s optimal, obj 15.540611, ~3.4s optimal, obj 15.540611, ~3.5s
gdpopt.lbb failed optimal, obj 12.089262, ~148.2s optimal, obj 12.089262, ~87.7s
gdpopt.ric maxTimeLimit, UB 13.275276, LB 4.0 maxTimeLimit, UB 13.275276, LB 4.0 maxTimeLimit, UB 80.634986, LB 4.0

The local gdpopt.lbb failure is solver routing rather than a Spectralog construction failure:

ValueError: GAMS writer passed solver (dicopt) unsuitable for model type (nlp)

The suspicious result is GLOA. It certifies 15.540611, while direct reformulations and LBB with Gurobi/BARON find 12.089262. This appears consistent with Pyomo/pyomo#3939, where gdpopt.gloa currently uses the concave MC++ slope for both concave and convex affine cuts.

I checked the installed Pyomo source in this environment and confirmed it has that exact ccSlope reuse pattern. I then ran a temporary control monkeypatch that uses ccSlope for the concave lower cut and cvSlope for the convex upper cut. With that patch, Spectralog GLOA no longer false-converged:

termination: maxTimeLimit
iterations: 127
lower_bound: 6.0
upper_bound: 12.08926220096831
relative_gap: 50.36918%
time: 300.16660s

So the Pyomo fix would not make this PR solve Spectralog globally with GLOA in 5 minutes, but it would make the result conservative: GLOA times out with the correct incumbent and an open gap instead of incorrectly reporting optimal at a worse solution. For this PR, I would treat the reformulation and LBB results as the reliable benchmark evidence, and flag unpatched GLOA certificates for this nonconvex instance as unreliable until Pyomo/pyomo#3939 is fixed.

@bernalde bernalde left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer review of head b36cb3f.

The bound derivation is valid: with R identity in the committed data, val[j] = sum_k r_k^2 >= 0, and with P in [0, 1000], A >= 0, the per-k corner maximum gives an upper bound over the whole variable box — a superset of the feasible region — so the bounds cannot cut off any feasible point. The removal of the spectralog,gdpopt.enumerate row from pr58_local.csv is campaign configuration, not committed results, benchmark_cases/ already exists on main, and the cited upstream Pyomo/pyomo#3953 exists and matches the stated cause (2^30 enumeration; 3 compounds x 10 wave numbers = 30 two-disjunct disjunctions checks out). The strengthened test_committed_pr58_local_cases_cover_default_matrix adds exact set equality on top of the count, strictly stronger than base.

Findings:

Nonblocking: bound helper assumptions — _val_bounds silently assumes R is identity and A, C >= 0; the sum-of-squares decomposition breaks if m.R ever gains off-diagonals. Also max_absorbance is loop-invariant in k (hoist it) and misnamed — it bounds the modeled signal sum P*A, not absorbance. Hoist, rename (e.g. max_modeled_signal), and state the identity-R assumption in a comment or assertion.

Nonblocking: vacuous assert — assert not any(... LogicalConstraint ...) in tests/test_spectralog.py is vacuously true: build_model() declares no LogicalConstraint (only BooleanVar + associate_binary_var). Drop the line or comment it as a forward guard; the Disjunction/Disjunct deactivation asserts are the real checks.

Question: optimum discrepancy — the module docstring (untouched) states "Optimal value: 12.0893" while this PR's GLOA run proves LB=UB=15.5406 on this implementation. Since the added bounds provably contain all feasible points, they cannot explain the gap. Is 12.0893 from the source papers' differing formulation/scaling, or is a subsolver bound wrong? Reconcile — either update the docstring with the verified optimum for this implementation (with provenance) or document the formulation difference.

Nonblocking: closes-vs-refs — issue #75's checklist leaves GLOA and Enumeration unchecked; this PR fixes GLOA and excludes Enumeration pending Pyomo/pyomo#3953. "Closes #75" is defensible only if the exclusion is accepted as the resolution; otherwise treat as Refs #75 and track re-adding the enumerate row once the upstream fix lands.

Sibling note: PR #145 also touches tests/test_benchmark.py; the hunks are disjoint and merge-tree reports no conflict.

Summary — Blocking: 0. Nonblocking: 3. Questions: 1. Tests: independent numeric re-derivation of all 8 bound constants (all match to 4 decimals); CI is green at this head (6/6 checks). Merge-ready once the optimum discrepancy and issue-linkage are resolved. Posting as COMMENT (author account); the main ruleset requires one approving review from another maintainer.

@bernalde

Copy link
Copy Markdown
Member Author

Addressed the review of head b36cb3f in commit a959a54:

  • val-bounds-assumptions: _val_bounds now documents the identity-R and nonnegative-data assumptions; the loop-invariant bound is hoisted out of the compound loop and renamed max_modeled_signal.
  • vacuous-logical-assert-147: kept as an explicitly commented forward guard.
  • docstring-optimum-mismatch (question) — resolved with a decisive local run: BARON on the Big-M reformulation of this head proves the global optimum is 12.089262 (LB = UB, optcr=1e-6, 120 s limit), so the module docstring's 12.0893 is correct and stays. The PR body's "GLOA optimal LB=UB=15.5406" is therefore not a global optimum: that run used --solver-profile gams-local, whose local NLP subsolvers void GLOA's global-bound guarantee, so GLOA's "optimal" label there certifies only its own converged bounds. The new variable bounds are unaffected — the 12.0893 optimum was found with them in place, confirming they cut nothing. Follow-up worth tracking on Refactor spectralog to fix benchmark issues #75: re-run the spectralog GLOA row with a global-subsolver profile (gams-baron) before checking the GLOA box.
  • closes-vs-refs-75: recorded here since the body stands — with GLOA's evidence now known to be non-global and Enumeration excluded pending GDPopt enumerate materializes all discrete solutions before checking time_limit Pyomo/pyomo#3953, the link should be read as Refs #75; if merged as-is, reopen Refactor spectralog to fix benchmark issues #75 for the GLOA re-run and the enumerate-row restoration.

Tests: pixi run pytest tests/test_spectralog.py tests/test_benchmark.py — 24 passed; black applied and clean. CI for the new head was starting when this was posted; the previous head was green (6/6).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor spectralog to fix benchmark issues

1 participant