Perf/int32#17
Open
FBumann wants to merge 121 commits into
Open
Conversation
linopy/constants.py — Added DEFAULT_LABEL_DTYPE = np.int32 linopy/model.py — Variable and constraint label assignment now uses np.arange(..., dtype=DEFAULT_LABEL_DTYPE) with overflow guards that raise ValueError if labels exceed int32 max. linopy/expressions.py — _term coord assignment and all .astype(int) for vars arrays now use DEFAULT_LABEL_DTYPE (int32). linopy/common.py — fill_missing_coords uses np.arange(..., dtype=DEFAULT_LABEL_DTYPE). Polars schema inference now checks array.dtype.itemsize instead of the old OS/numpy-version hack. test/test_constraints.py — Updated 2 dtype assertions to use np.issubdtype instead of == int. test/test_dtypes.py (new) — 7 tests covering int32 labels, expression vars, solve correctness, and overflow guards.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…k to int64 via astype(int), now use DEFAULT_LABEL_DTYPE. Also Variables.to_dataframe arange for map_labels. - linopy/constraints.py: Constraints.to_dataframe arange for map_labels. - linopy/common.py: save_join outer-join fallback was casting to int64.
* Fix multiplication of constant-only LinearExpression When multiplying a constant-only LinearExpression with another expression, the code would fail with IndexError when trying to access _term=0 on an empty term dimension. The fix correctly returns a LinearExpression (not QuadraticExpression) since multiplying by a constant preserves linearity. * fix: add type casts for mypy * fix: use cast instead of isinstance for runtime type check * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…ons on small) (PyPSA#564) * perf: use Polars streaming engine for LP file writing Extract _format_and_write() helper that uses lazy().collect(engine="streaming") with automatic fallback, replacing 7 instances of df.select(concat_str(...)).write_csv(...). * fix: log warning with traceback when Polars streaming fallback triggers * perf: speed up LP constraint writing by replacing concat+sort with join Replace the vertical concat + sort approach in Constraint.to_polars() with an inner join, so every row has all columns populated. This removes the need for the group_by validation step in constraints_to_file() and simplifies the formatting expressions by eliminating null checks on coeffs/vars columns. * fix: missing space in lp file * perf: skip group_terms when unnecessary and avoid xarray broadcast for short DataFrame - Skip group_terms_polars when _term dim size is 1 (no duplicate vars) - Build the short DataFrame (labels, rhs, sign) directly with numpy instead of going through xarray.broadcast + to_polars - Add sign column via pl.lit when uniform (common case), avoiding costly numpy string array → polars conversion Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * perf: skip group_terms in LinearExpression.to_polars when no duplicate vars Check n_unique before running the expensive group_by+sum. When all variable references are unique (common case for objectives), this saves ~31ms per 320k terms. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * perf: reduce per-constraint overhead in Constraint.to_polars() Replace np.unique with faster numpy equality check for sign uniformity. Eliminate redundant filter_nulls_polars and check_has_nulls_polars on the short DataFrame by applying the labels mask directly during construction. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: handle empty constraint slices in sign_flat check Guard against IndexError when sign_flat is empty (no valid labels) by checking len(sign_flat) > 0 before accessing sign_flat[0]. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: add LP write speed improvement to release notes Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * bench: add LP write benchmark script with plotting * bench: larger model * perf: Add maybe_group_terms_polars() helper in common.py that checks for duplicate (labels, vars) pairs before calling group_terms_polars. Use it in both Constraint.to_polars() and LinearExpression.to_polars() to avoid expensive group_by when terms already reference distinct variables * Add variance to plot * test: add coverage for streaming fallback and maybe_group_terms_polars * fix: mypy * fix: mypy * Move kwargs into method for readability * Remove fallback and pin polars >=1.31 * Remove the benchmark_lp_writer.py --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Add auto mask option to model.py * Also capture rhs * Add benchmark_auto_mask.py * Use faster numpy operation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ruff and release notes * Optimize mask application and null expression check Performance improvements: - Use np.where() instead of xarray where() for mask application (~38x faster) - Use max() == -1 instead of all() == -1 for null expression check (~30% faster) These optimizations make auto_mask have minimal overhead compared to manual masking. * Fix mask broadcasting for numpy where in add_constraints The switch from xarray's where() to numpy's where() broke dimension-aware broadcasting. A 1D mask with shape (10,) was being broadcast to (1, 10) instead of (10, 1), applying to the wrong dimension. Fix: Explicitly broadcast mask to match data.labels shape before using np.where. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
* add dummy text * change linopy version discovery * remove redundnat comments --------- Co-authored-by: Robbie Muir <robbie.muir@gmail.com>
…yPSA#575) * fix: revert np.where to xarray.where * trigger
* reinsert broadcasting of masks * update release notes * consolidate broadcast mask into new function, add tests for subsets * align test logic to broadcasting * Reinsert broadcasted mask (PyPSA#581) * 1. Moved the dimension subset check into broadcast_mask 2. Added a brief docstring to broadcast_mask * Add tests for superset dims --------- Co-authored-by: FBumann <117816358+FBumann@users.noreply.github.com>
Replace dead maths.ed.ac.uk links with highs.dev and correct options URL. Use "HiGHS" consistently in docstrings.
* Add Knitro solver support - Add Knitro detection to available_solvers list - Implement Knitro solver class with MPS/LP file support - Add solver capabilities for Knitro (quadratic, LP names, no solution file) - Add tests for Knitro solver functionality - Map Knitro status codes to linopy Status system * Fix Knitro solver integration * Document Knitro and improve file loading * code: add check to solve mypy issue * code: remove unnecessary candidate loaders * code: remove unnecessary candidate loaders * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * code: use just KN_read_problem for lp * add read_options * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * code: update KN_read_problem calling * code: new changes from Daniele Lerede * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * code: add reported runtime * code: remove unnecessary code * doc: update README.md and realease_notes * code: add new unit tests for Knitro * code: add new unit tests for Knitro * code: add test for lp for knitro * code: add test for lp for knitro * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * code: add-back again skip * code: remove uncomment to skipif * add namedtuple * include pre-commit checks * fix type checking * simplify Knitro solver class Remove excessive error handling, getattr usage, and unpack_value_and_rc. Use direct Knitro API calls, extract _set_option and _extract_values helpers. Add missing INTEGER_VARIABLES and READ_MODEL_FROM_FILE capabilities. Fix test variable names and remove dead warmstart/basis no-ops. * code: update pyproject.toml and solver attributes * code: update KN attribute dependence * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Fabrizio Finozzi <fabrizio.finozzi.business@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…sos features (PyPSA#549) * The SOS constraint reformulation feature has been implemented successfully. Here's a summary: Implementation Summary New File: linopy/sos_reformulation.py Core reformulation functions: - validate_bounds_for_reformulation() - Validates that variables have finite bounds - compute_big_m_values() - Computes Big-M values from variable bounds - reformulate_sos1() - Reformulates SOS1 constraints using binary indicators and Big-M constraints - reformulate_sos2() - Reformulates SOS2 constraints using segment indicators and adjacency constraints - reformulate_all_sos() - Reformulates all SOS constraints in a model Modified: linopy/model.py - Added import for reformulate_all_sos - Added reformulate_sos_constraints() method to Model class - Added reformulate_sos: bool = False parameter to solve() method - Updated SOS constraint check to automatically reformulate when reformulate_sos=True and solver doesn't support SOS natively New Test File: test/test_sos_reformulation.py 36 comprehensive tests covering: - Bound validation (finite/infinite) - Big-M computation - SOS1 reformulation (basic, negative bounds, multi-dimensional) - SOS2 reformulation (basic, trivial cases, adjacency) - Integration with solve() and HiGHS - Equivalence with native Gurobi SOS support - Edge cases (zero bounds, multiple SOS, custom prefix) Usage Example m = linopy.Model() x = m.add_variables(lower=0, upper=1, coords=[pd.Index([0, 1, 2], name='i')], name='x') m.add_sos_constraints(x, sos_type=1, sos_dim='i') m.add_objective(x.sum(), sense='max') # Works with HiGHS (which doesn't support SOS natively) m.solve(solver_name='highs', reformulate_sos=True) * Documentation Summary New Section: "SOS Reformulation for Unsupported Solvers" Added a comprehensive section (~300 lines) covering: 1. Enabling Reformulation - Shows reformulate_sos=True parameter and manual reformulate_sos_constraints() method 2. Requirements - Explains finite bounds requirement for Big-M method 3. Mathematical Formulation - Clear LaTeX math for both: - SOS1: Binary indicators y_i, upper/lower linking constraints, cardinality constraint - SOS2: Segment indicators z_j, first/middle/last element constraints, cardinality constraint 4. Interpretation - Explains how the constraints work intuitively with examples 5. Auxiliary Variables and Constraints - Documents the naming convention (_sos_reform_ prefix) 6. Multi-dimensional Variables - Shows how broadcasting works 7. Edge Cases Table - Lists all handled edge cases (single-element, zero bounds, all-positive, etc.) 8. Performance Considerations - Trade-offs between native SOS and reformulation 9. Complete Example - Piecewise linear approximation of x² with HiGHS 10. API Reference - Added method signatures for: - Model.add_sos_constraints() - Model.remove_sos_constraints() - Model.reformulate_sos_constraints() - Variables.sos property * Added Tests for Multi-dimensional SOS Unit Tests - test_sos2_multidimensional: Tests that SOS2 reformulation with multi-dimensional variables (i, j) correctly creates: - Segment indicators z with shape (i: n-1, j: m) - Cardinality constraint preserves the j dimension Integration Tests - test_multidimensional_sos2_with_highs: Solves a multi-dimensional SOS2 problem with HiGHS and verifies: - Optimal objective value (4 total - two adjacent non-zeros per column) - SOS2 constraint satisfied for each j: at most 2 non-zeros, and if 2, they're adjacent Test Results test_sos1_multidimensional PASSED test_sos2_multidimensional PASSED test_multidimensional_sos1_with_highs PASSED test_multidimensional_sos2_with_highs PASSED The implementation correctly handles multi-dimensional variables by leveraging xarray's broadcasting - the SOS constraint is applied along the sos_dim for each combination of the other dimensions. * Add custom big_m parameter for SOS reformulation Allow users to specify custom Big-M values in add_sos_constraints() for tighter LP relaxations when variable bounds are conservative. - Add big_m parameter: scalar or tuple(upper, lower) - Store as variable attrs (big_m_upper, big_m_lower) - Skip bound validation when custom big_m provided - Scalar-only design ensures NetCDF persistence works correctly For per-element Big-M values, users should adjust variable bounds directly. * Add custom big_m parameter for SOS reformulation Allow users to specify custom Big-M values in add_sos_constraints() for tighter LP relaxations when variable bounds are conservative. - Add big_m parameter: scalar or tuple(upper, lower) - Store as variable attrs (big_m_upper, big_m_lower) for NetCDF persistence - Use tighter of big_m and variable bounds: min() for upper, max() for lower - Skip bound validation when custom big_m provided (allows infinite bounds) Scalar-only design ensures NetCDF persistence works correctly. For per-element Big-M values, users should adjust variable bounds directly. * Simplification summary: ┌──────────────────────┬───────────┬───────────┬───────────┐ │ File │ Before │ After │ Reduction │ ├──────────────────────┼───────────┼───────────┼───────────┤ │ sos_reformulation.py │ 377 lines │ 223 lines │ 41% │ ├──────────────────────┼───────────┼───────────┼───────────┤ │ sos-constraints.rst │ 647 lines │ 164 lines │ 75% │ └──────────────────────┴───────────┴───────────┴───────────┘ Code changes: - Merged validate_bounds_for_reformulation into compute_big_m_values - Factored out add_linking_constraints helper in SOS2 - Used np.minimum/np.maximum instead of xr.where - Kept proper docstrings with Parameters/Returns sections Doc changes: - Removed: Variable Representation, LP File Export, Common Patterns, Performance Considerations - Trimmed: Examples to one each, Mathematical formulation to equations only - Condensed: API reference, multi-dimensional explanation * Revert some docs changes to be more surgical * Add math to docs * Improve docs * Code simplifications: 1. sos_reformulation.py (230 → 203 lines): - compute_big_m_values now returns single DataArray (not tuple) - Removed all lower bound handling - only supports non-negative variables - Removed add_linking_constraints helper function - Simplified SOS1/SOS2 to only add upper linking constraints 2. model.py: - Simplified big_m parameter from float | tuple[float, float] | None to float | None - Removed big_m_lower attribute handling 3. Documentation (sos-constraints.rst): - Updated big_m type signature - Removed asymmetric Big-M example - Added explicit requirement that variables must have non-negative lower bounds 4. Tests (46 → 38 tests): - Removed tests for negative bounds - Removed tests for tuple big_m - Added tests for negative lower bound validation error Rationale: The mathematical formulation in the docs assumes x ∈ ℝⁿ₊ (non-negative reals). This matches 99%+ of SOS use cases (selection indicators, piecewise linear weights). The simplified code is now consistent with the documented formulation. * Fix mypy * Fix mypy * Add constants for sos attr keys * Add release notes * Fix SOS reformulation: undo after solve, validate big_m, vectorize - solve() now undoes SOS reformulation after solving, preserving model state - Validate big_m > 0 in add_sos_constraints (fail fast) - Vectorize SOS2 middle constraints, eliminate duplicate compute_big_m_values - Warn when reformulate_sos=True is ignored for SOS-capable solvers - Add tests for model immutability, double solve, big_m validation, undo * tiny refac, plus uncovered test * refac: move reformulating function to module * Fix SOS reformulation: rollback, skipped attrs, undo in solve, sort coords - Remove SOS attrs for skipped variables (size<=1, M==0) so solvers don't see them as SOS constraints - Wrap reformulation loop in try/except for transactional rollback - Move undo into finally block in Model.solve() for exception safety - Sort variables by coord values before building adjacency constraints to match native SOS weight-based ordering * update release notes [skip ci] --------- Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
…tive) (PyPSA#576) * feat: add piecewise linear constraint API Add `add_piecewise_constraint` method to Model class that creates piecewise linear constraints using SOS2 formulation. Features: - Single Variable or LinearExpression support - Dict of Variables/Expressions for linking multiple quantities - Auto-detection of link_dim from breakpoints coordinates - NaN-based masking with skip_nan_check option for performance - Counter-based name generation for efficiency The SOS2 formulation creates: 1. Lambda variables with bounds [0, 1] for each breakpoint 2. SOS2 constraint ensuring at most two adjacent lambdas are non-zero 3. Convexity constraint: sum(lambda) = 1 4. Linking constraints: expr = sum(lambda * breakpoints) * Fix lambda coords * rename to add_piecewise_constraints * rename to add_piecewise_constraints * fix types (mypy) * linopy/constants.py — Added PWL_DELTA_SUFFIX = "_delta" and PWL_FILL_SUFFIX = "_fill". linopy/model.py — - Added method: str = "sos2" parameter to add_piecewise_constraints() - Updated docstring with the new parameter and incremental formulation notes - Refactored: extracted _add_pwl_sos2() (existing SOS2 logic) and added _add_pwl_incremental() (new delta formulation) - Added _check_strict_monotonicity() static method - method="auto" checks monotonicity and picks accordingly - Numeric coordinate validation only enforced for SOS2 test/test_piecewise_constraints.py — Added TestIncrementalFormulation (10 tests) covering: single variable, two breakpoints, dict case, non-monotonic error, decreasing monotonic, auto-select incremental/sos2, invalid method, extra coordinates. Added TestIncrementalSolverIntegration (Gurobi-gated). * 1. Step sizes: replaced manual loop + xr.concat with breakpoints.diff(dim).rename() 2. Filling-order constraints: replaced per-segment individual add_constraints calls with a single vectorized constraint via xr.concat + LinearExpression 3. Mask computation: replaced loop over segments with vectorized slice + rename 4. Coordinate lists: unified extra_coords/lambda_coords — lambda_coords = extra_coords + [bp_dim_index], eliminating duplicate list comprehensions * rewrite filling order constraint * Fix monotonicity check * Summary Files Modified 1. linopy/constants.py — Added 3 constants: - PWL_BINARY_SUFFIX = "_binary" - PWL_SELECT_SUFFIX = "_select" - DEFAULT_SEGMENT_DIM = "segment" 2. linopy/model.py — Three changes: - Updated imports to include the new constants - Updated _resolve_pwl_link_dim with an optional exclude_dims parameter (backward-compatible) so auto-detection skips both dim and segment_dim - Added _add_dpwl_sos2 private method implementing the disaggregated convex combination formulation (binary indicators, per-segment SOS2 lambdas, convexity, and linking constraints) - Added add_disjunctive_piecewise_constraints public method with full validation, mask computation, and dispatch 3. test/test_piecewise_constraints.py — Added 7 test classes with 17 tests: - TestDisjunctiveBasicSingleVariable (3 tests) — equal segments, NaN padding, single-breakpoint segments - TestDisjunctiveDictOfVariables (2 tests) — dict with segments, auto-detect link_dim - TestDisjunctiveExtraDimensions (1 test) — extra generator dimension - TestDisjunctiveValidationErrors (5 tests) — missing dim, missing segment_dim, same dim/segment_dim, non-numeric coords, invalid expr - TestDisjunctiveNameGeneration (2 tests) — shared counter, custom name - TestDisjunctiveLPFileOutput (1 test) — LP file contains SOS2 + binary sections - TestDisjunctiveSolverIntegration (3 tests) — min/max picks correct segment, dict case with solver * docs: add piecewise linear constraints documentation Create dedicated documentation page covering all three PWL formulations: SOS2 (convex combination), incremental (delta), and disjunctive (disaggregated convex combination). Includes math formulations, usage examples, comparison table, generated variables reference, and solver compatibility. Update index.rst, api.rst, and sos-constraints.rst. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: improve disjunctive piecewise linear test coverage Add 17 new tests covering masking details, expression inputs, multi-dimensional cases, multi-breakpoint segments, and parametrized multi-solver testing. Disjunctive tests go from 17 to 34 unique methods. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: Add notebook to showcase piecewise linear constraint * Add cross reference to notebook * Improve notebook * docs: add release notes and cross-reference for PWL constraints Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix mypy issue in test * Improve docs about incremental * refactor and add tests * fix: reject non-trailing NaN in incremental piecewise formulation Validate that NaN breakpoints are trailing-only along dim. For method='incremental', raise ValueError on gaps. For method='auto', fall back to SOS2 instead. Add _has_trailing_nan_only helper. * further refactor * extract piecewise linear logic into linopy/piecewise.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: allow broadcasted mask * fix merge conflict in release notes * refactor: remove link_dim from piecewise constraint API The linking dimension is now always auto-detected from breakpoint coordinates matching the expression dict keys, simplifying the public API of add_piecewise_constraints and add_disjunctive_piecewise_constraints. * refactor: use LinExprLike type alias and consolidate piecewise validation Extract _validate_piecewise_expr helper to replace duplicated isinstance checks in _auto_broadcast_breakpoints and _resolve_expr. Add LinExprLike type alias to types.py. Update docs, tests, and breakpoints factory. * fix: resolve mypy errors in piecewise module * update release notes [skip ci] --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
* feat: add reformulate_sos='auto' support to solve() - Accept 'auto' as string literal in reformulate_sos parameter (line 1230) - When reformulate_sos='auto' and solver lacks SOS support, silently reformulate - When reformulate_sos='auto' and solver supports SOS natively, pass through without warning - Update error message to mention both True and 'auto' options (line 1424) - Add comprehensive test suite with 5 new test cases covering all scenarios - All 57 SOS reformulation tests pass * fix: improve reformulate_sos validation, DRY up branching, strengthen tests Validate reformulate_sos input early, collapse duplicate True/auto branches, fix docstring type notation, add tests for invalid values and no-SOS no-op, strengthen SOS2 test to actually verify adjacency constraint enforcement. * fix: resolve mypy errors in piecewise and SOS reformulation tests Widen segment types from list[list[float]] to list[Sequence[float]] and add missing type annotations in test fixtures.
…PSA#589) Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
Bumps the github-actions group with 3 updates: [actions/download-artifact](https://github.com/actions/download-artifact), [actions/upload-artifact](https://github.com/actions/upload-artifact) and [crazy-max/ghaction-chocolatey](https://github.com/crazy-max/ghaction-chocolatey). Updates `actions/download-artifact` from 7 to 8 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v7...v8) Updates `actions/upload-artifact` from 6 to 7 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v6...v7) Updates `crazy-max/ghaction-chocolatey` from 3 to 4 - [Release notes](https://github.com/crazy-max/ghaction-chocolatey/releases) - [Commits](crazy-max/ghaction-chocolatey@v3...v4) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: crazy-max/ghaction-chocolatey dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* code: expose knitro context and modify _extract_values * doc: update release_notes.rst * code: include pre-commit checks
* enable quadratic for win with scip * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add release note * Drop reference to SCIP bug --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
…yPSA#746) Follow-up to PyPSA#739 (Python 3.11 syntax): SolverStatus.from_termination_condition now returns `Self` instead of the quoted "SolverStatus", matching its sibling `process` and Status.from_termination_condition, which PyPSA#739 converted but missed here. No functional change. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add AGENTS.md as the single source of truth for how AI-generated content in PRs, issues and review discussions must be marked, and what the human author is expected to write by hand. Stop ignoring AGENTS.md so it is shared with contributors. Point doc/contributing.rst, CLAUDE.md and the PR template at it. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…PSA#762) * fix(alignment): tolerate xarray without CoordinateValidationError CoordinateValidationError was only added in xarray 2025.6.0, but linopy declares a floor of xarray>=2024.2.0. The hard import made `import linopy` fail with ImportError on any xarray in [2024.2.0, 2025.5]. Fall back to ValueError when the symbol is absent. CoordinateValidationError subclasses ValueError, so the `except (ValueError, CoordinateValidationError)` in broadcast_to_coords is unchanged in behaviour on older xarray. Fixes PyPSA#761 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(alignment): trim inline comment on the CoordinateValidationError fallback Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…es) (PyPSA#766) * fix: read tuple coords entries as xarray's (dim_name, values) _coords_to_dict treated a tuple coords entry as a bare value sequence (like a list), which broke the standard xarray form ``coords=[("origin", origins)]`` — a scalar bound against such coords raised an inscrutable "inhomogeneous shape" error from numpy. v0.7.0 delegated to xarray and supported this form; the PyPSA#742 alignment refactor regressed it (unreleased) and pinned the value-sequence reading in tests. Restore xarray's convention: a tuple entry is ``(dims, data[, attrs])`` (first element names the dimension), while a bare value sequence uses a list. Length-1 / scalar-value tuples raise a clear convention error, as xarray does. Also name unnamed pd.Index entries on a copy so an entry's .name matches its dim consistently across forms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: parameterize and reorganize alignment coords tests Collapse TestCoordsToDict's per-section single-assertion tests into a few form-parameterized tests (one table for naming forms, one for the skipped unlabeled forms, one folding the five scattered "entry raises TypeError" cases), dropping the # -- section comments in favour of the parametrization. Move the end-to-end add_variables checks out of TestCoordsToDict into a new TestAddVariablesCoords class, since they exercise the model wiring rather than _coords_to_dict, and fold them into a single parametrized test. No behaviour change and no coverage lost. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: correct coords-entry TypeError to not list tuple as a bare sequence The fallback message in _coords_to_dict still advertised `tuple` as a valid "unnamed sequence", which contradicts the (dim_name, values) tuple handling — a user following it would hit the tuple-convention error instead. Name the accurate forms (pd.Index / unlabeled list-range-ndarray / (dim_name, values) tuple). Adds a scalar-value tuple case to the parametrized raises table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch test assertions from the internal `.data` accessor to the public `.coords`/`.indexes`/`.sizes` properties (thin pass-throughs to `.data`, so behaviour is identical). Genuine `_factor` dimension checks in test_quadratic_expression stay on the internal array, but now use `.vars.sizes[FACTOR_DIM]` — the `_factor` helper dim lives on `vars`, not `coeffs`, and isn't surfaced by any public accessor. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… grouping by names (PyPSA#750, PyPSA#753) (PyPSA#751) * fix(groupby): group by the name of a non-dimension coordinate `LinearExpression.groupby` could not group by an attached non-dimension coordinate. `expr.groupby("period").sum()` raised `ValueError: period already exists as coordinate or variable name`, and passing the coordinate `DataArray` (`groupby(period)`) raised because the fast path dropped only the dimension index, then renamed the group dim onto a name still held by the attached coordinate. Fix both paths: - `sum()` resolves a string group naming an existing coordinate to that coordinate so it takes the fast path, and drops every coordinate aligned to the grouped dimension (index, MultiIndex levels, auxiliary coords) before reshaping, since collapsing the dimension invalidates them all. - The `groupby` property detaches an attached non-dimension coordinate used as the group before handing it to xarray, so xarray does not try to re-expand it when recombining groups (the `use_fallback=True` path). `expr.groupby("period").sum()` now mirrors `xarray.Dataset.groupby`. Closes PyPSA#750 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(groupby): assert grouping against hard-coded results Reorganize the PyPSA#750 coverage into TestGroupbyByAttachedCoordinate, a parametrized matrix that asserts the grouped expression against hard-coded `vars`/`coeffs` literals (on a deterministic 4- and 8-variable model) instead of comparing to a sibling computation that could share the same bug. Covers single-key (name / DataArray x use_fallback), multi-key (list / tuple x use_fallback), an extra auxiliary coord on the grouped dimension, and a 2-D variable that must keep its other dimension. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(groupby): support single-element key list; document multi-key contract A single-element key list (`groupby(["period"])`) now groups like the scalar key, matching xarray -- it is unwrapped in both `sum()` and the `groupby` property. Multi-key grouping must be spelled with names (`["period", "season"]`); a list of `DataArray`s is unhashable and raises in xarray itself, so linopy mirrors that (covered by an explicit `pytest.raises` row in the matrix). Also shorten the test class docstring to house style. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: make groupby PyPSA#750 release note concise and user-facing Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(groupby): don't detach a MultiIndex level in the fallback path The GH PyPSA#750 detach must only drop *free* (non-indexed) coordinates. The earlier change also dropped a MultiIndex level when grouping by it via `use_fallback=True`, leaving the dimension without an index (`('snapshot',) are not coordinates with an index`). Guard the detach with `group.name not in data.xindexes` so MultiIndex levels are left intact. Grouping by a MultiIndex level now works on both paths (the pydata/xarray 6836 case, fixed upstream). Add a parametrized regression test over both levels and both paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(groupby): extract _resolve_group helper; address review - Extract the single-element-list unwrap + coordinate-name resolution shared by `sum()` and the `groupby` property into one `_resolve_group` helper, removing the duplication (and the drift between the two that caused the earlier MultiIndex-level regression). - Drop the now-stale `(GH PyPSA#750)` references from code comments; the link lives in the release notes. - Add a test for grouping by a dimension coordinate name (the fast-path broadening), and note it in the release note. - Simplify `test_multi_key`: a multi-key group always uses the xarray fallback, so drop the redundant `use_fallback` parametrization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: scope groupby PyPSA#750 note to non-dimension coordinates Grouping by a dimension coordinate or a MultiIndex level by name already worked; only the non-dimension (free) coordinate case was broken. Correct the release note, which had overclaimed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(groupby): trim inline comments to one-liners Drop the verbose comment blocks; the rationale lives in the _resolve_group docstring and the regression tests enforce the invariants (e.g. test_multiindex_level guards the xindexes detach guard). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(groupby): fast path for multi-key groupby([names]) `groupby(["a","b"]).sum()` previously dropped to the slow xarray fallback. Resolve a list of coordinate names (1-D, same dim) to a value frame so it rides the existing reindex fast path, then unstack the stacked result back into one dimension per key -- byte-identical to the fallback, sparse fill cells included. The DataFrame grouper is untouched and stays compact (stacked MultiIndex over observed combinations only), so this is non-breaking. One dimension per key is a dense cartesian grid, so a sparse key crossing materialises mostly-fill cells. Warn (pointing at the DataFrame grouper) when the grid is much larger than the observed combinations; the check reads the collapsed MultiIndex levels, so it is O(observed) and fires before unstack allocates. See PyPSA#753; sparse-representation follow-ups tracked against PyPSA#740. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(groupby): add observed= to sum(); extract multikey helpers * docs: release note for groupby observed= feature --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Fabian <fab.hof@gmx.de>
* fix: align Variable.fix() value to the variable's coordinates fix() converted the value with as_dataarray().broadcast_like(self.labels), which aligns only by dimension name and so worked solely for the default `dim_0`. On a named dimension, a positional value (list/array) gained a spurious `dim_0` and broadcast across the real dimension instead of onto it, silently building a wrong fix constraint (one fixing every entry to every value). Use broadcast_to_coords against the variable's own coords — the same coords- aware alignment add_variables uses for lower/upper: scalars broadcast, positional inputs land on the right dimension, named pandas/xarray inputs align by coordinate value, and a mismatch raises an error naming the variable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: parametrize fix() value alignment over dtypes and rejection cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix types * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix variables via bound collapse instead of equality constraint (PyPSA#773) * feat: fix variables via bound collapse, honoring bounds at export Variable.fix() now collapses lower = upper = value (the JuMP/Pyomo/gurobipy meaning of "fix") instead of adding a __fix__ equality constraint. Pre-fix bounds are stashed as _stashed_lower / _stashed_upper data variables on the variable's own Dataset, so unfix() restores them and the state round-trips through netcdf for free. Fixing is now a pure value change, so fix/re-solve loops stay on the persistent in-place update path instead of forcing a solver rebuild, and model.constraints is no longer polluted with __fix__ entries. A fix value outside the current bounds warns and overrides; fixing a binary to anything other than 0/1 raises. Removes FIX_CONSTRAINT_PREFIX and the __fix__ cleanup in remove_variables. Honoring a fixed binary that stays binary required fixing binary-bound export, which several paths hardcoded to [0, 1]: the LP writer now emits explicit bounds for fixed binaries, and the HiGHS and Mosek direct backends no longer override binary bounds (they already come from M.lb/M.ub). This also fixes the latent bug that a binary's bounds could not be restricted at all. Tests: bound-collapse unit coverage in test_fix_relax.py, and a cross-solver, cross-io integration test (test_fixed_variable_is_held) asserting the fix is honored when solving for continuous, integer and binary, in both bound directions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(fix): strict binary validation, no stash-var leak in flat, generalized STASHED_ATTRS fix() rejects non-0/1 binary values (np.isclose) before rounding; flat/to_polars drop internal stash vars; bound checks use direct attrs subscript and hoisted locals. * refact comments and tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Fabian <fab.hof@gmx.de> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: remove unnneded import in test file * test: assert fix() collapses bounds instead of adding a constraint --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Fabian <fab.hof@gmx.de> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add indicator constraints * Phase A: indicator fields in unified constraint data model Add is_indicator/binary_var/binary_val to ConstraintBase, Constraint, and CSRConstraint (slots, init, from_mutable, data, netcdf round-trip). Add Constraints.indicator/.regular container accessors. * feat(indicator): unify indicator constraints into normal container (model+common+matrices) * feat(indicator): route LP write and model copy through unified container (io) * feat(indicator): consume MatrixAccessor indicator arrays in Gurobi (solvers) * test(indicator): repair assertions for unified container semantics * fix(indicator): satisfy mypy on return type and binval cast * fix(gurobi): key indicator sense_map on single-char senses MatrixAccessor.indicator_sense holds single-char senses (<, >, =) from to_matrix_with_rhs; the direct-API sense_map was keyed on two-char strings, raising KeyError on the direct path (LP-file path was fine). * Phase C: tests for unified indicator constraints Add freeze round-trip, netCDF round-trip (mutable + frozen), matrix split, copy preservation, and direct-API Gurobi solve coverage. * Exclude indicators from Constraints.ncons and to_matrix (plan §4) ncons feeds model.ncons which the Mosek backend uses to size the (regular-only) matrix M.A; keep them aligned. Also exclude indicators from the consolidated to_matrix. Add ncons assertion and release note. * fix mypy in new tests * refactor tests * refactor: DRY constraint add paths, matrix assembly, and copy - share name/dispatch/label helpers between add_constraints and add_indicator_constraints - collapse duplicated assembly in MatrixAccessor._build_cons - add ConstraintBase.data_attrs; use it in copy() --------- Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
* Added m.copy() method. * Added testing suite for m.copy(). * Fix solver_dir type annotation. * Bug fix: xarray copyies need to be . * Intial commit: Working dualizer for LPs. * Fixed typing issues. Initialise dual objective with LinearExpression(None) instead of 0. * Fixed more typing issues. * Added code-block to dualize() examples. * Fix more typing issues. * Fixed variable bounds for dualisation of max problems (primal) --> reverse sign. * Updated docstrings for max -> min. * Moved copy to io.py, added deep-copy to all xarray operations. * Improved copy method: Strengtheninc copy protocol compatibility, check for deep copy independence. * Added release notes. * Made Model.copy defaulting to deep copy more explicit. * Fine-tuned docs and added to read the docs api.rst. * Clean up some methods in `dual` module (PyPSA#629) * Clean up some methods in `dual` module * Typing fix (L434). * Ignore type assessment in expressions.py --------- Co-authored-by: Bobby Xiong <bobbyxng@gmail.com> Co-authored-by: Lukas Trippe <lkstrp@pm.me> * fix: use dimension coords only when adding dual variables to avoid scalar coord conflicts. * fix CI testing issues * refactor dualisation, update tests, modularise functions and vectorise code * rename to _lift_bounds_to_constraints and make private * refactor, improve docstrings * improve readability, docstring formats, testing suite * fix typing in pytests * fix typing in pytests * improve test coverage and remove non-supported objective constant * Fix cases where constraints and variables of the same block/name can have different signs and/or bounds. Add tests * revert test-models.yml environments to upstream, see PR 522 * add release notes and improve docstrings of m.dualize() * docstring improvements * Rename dual.py to dualization.py; fix array coords, strip comments, dedupe lookups Use con.indexes[dim] (pd.Index) for dual variable coords so dualizing dimensioned models no longer raises TypeError. Collapse the duplicated bounded-gather and bound-lifting blocks into shared helpers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Annotate _gather_with_default result to satisfy mypy --------- Co-authored-by: Bryn Pickering <17178478+brynpickering@users.noreply.github.com> Co-authored-by: Lukas Trippe <lkstrp@pm.me> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* docs: cut release notes for v0.8.0 Promote the accumulated "Upcoming Version" entries to a finalized "Version 0.8.0" section and leave a fresh empty "Upcoming Version" placeholder for post-0.8.0 work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: condense and fix formatting in v0.8.0 release notes - Drop the duplicated MultiIndex-level-projection deprecation text from the Bug Fixes entry; keep the behavior there and cross-reference the Deprecations entry (and vice versa). - Fix a missing bullet on the to_gurobipy/to_highspy Internal entry that was rendering glued onto the previous item. - Fold the two lead bullets (variable_names/where, has_terms) into the Features section instead of orphaning them above the header. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: flag silent behavior changes in v0.8.0 bug fixes Two "Bug Fixes" change results without raising or warning, so users who relied on the old behavior get silently different models: - partial pandas bounds/masks are now broadcast across a missing dim instead of being dropped (a previously-ignored bound now applies); - Mosek may return a different (better-status) solution for the same model. Prefix both with a "⚠ Behavior change" marker and spell out the upgrade impact inline. No code change — behavior stays as merged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reorder v0.8.0 entries by importance (no wording changes) Pure reordering within sections so the highest-impact entries lead; bullet text is unchanged: - Features: CSR storage first, then solver inspection / framework / capabilities / licenses; minor expression props moved to a trailing "Other additions" group. - Performance: broad wins (direct comm, solution writeback) before the solver-specific Xpress entry. - Bug Fixes: both behavior-change entries surfaced to the top. - Breaking Changes: solution-array and read-only-property changes plus the Python 3.10 drop ahead of the coords edge cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
PyPSA#779) The docs pinned sphinx_book_theme==1.1.3, which has no upper bound on pydata-sphinx-theme and therefore resolved to pydata 0.16.1. The 1.1.x book theme predates pydata 0.16 and its mobile sidebar toggle JS breaks against it, so the navigation sidebar and ToC no longer open on mobile. Bump to sphinx_book_theme==1.2.0, the first release that pins and supports pydata-sphinx-theme==0.16.1, restoring the mobile toggles. Closes PyPSA#775 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(io): write LP bounds for tightened binary variables Per-element bounds set below the implied [0, 1] on a binary variable (e.g. masking out entries with upper = 0) were silently dropped by the LP file export: binaries appeared only in the `binary` section. The direct API honored them, so the same model relaxed when solved through io_api="lp". bounds_to_file now emits a bounds row for any binary whose bounds differ from (0, 1) anywhere, matching the direct path. Closes PyPSA#776 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: tone down binary-bounds release note Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: group LP binary-bounds tests into a class with a shared factory fixture * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test: annotate TestLPBinaryBounds fixture and tests for mypy --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Fabian <fab.hof@gmx.de> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
) * fix(io): write LP bounds for tightened binary variables Per-element bounds set below the implied [0, 1] on a binary variable (e.g. masking out entries with upper = 0) were silently dropped by the LP file export: binaries appeared only in the `binary` section. The direct API honored them, so the same model relaxed when solved through io_api="lp". bounds_to_file now emits a bounds row for any binary whose bounds differ from (0, 1) anywhere, matching the direct path. Closes PyPSA#776 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: tone down binary-bounds release note Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(model): allow {0,1} bounds for binaries in add_variables Binary bounds could previously only be set via the .lower/.upper setters after creation; add_variables(binary=True, ...) raised on any lower/upper. Now it accepts bounds as long as every value is 0 or 1 (unset bounds still default to 0/1), and validates the rest. Refs PyPSA#776 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: group LP binary-bounds tests into a class with a shared factory fixture * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test: add type annotations to binary bounds test params --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Fabian <fab.hof@gmx.de> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Bumps [matplotlib](https://github.com/matplotlib/matplotlib) from 3.9.1 to 3.11.0. - [Release notes](https://github.com/matplotlib/matplotlib/releases) - [Commits](matplotlib/matplotlib@v3.9.1...v3.11.0) --- updated-dependencies: - dependency-name: matplotlib dependency-version: 3.11.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the github-actions group with 2 updates: [actions/checkout](https://github.com/actions/checkout) and [codecov/codecov-action](https://github.com/codecov/codecov-action). Updates `actions/checkout` from 6 to 7 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6...v7) Updates `codecov/codecov-action` from 6 to 7 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: codecov/codecov-action dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 7.3.7 to 9.0.4. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/v9.0.4/CHANGES.rst) - [Commits](sphinx-doc/sphinx@v7.3.7...v9.0.4) --- updated-dependencies: - dependency-name: sphinx dependency-version: 9.0.4 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the benchmark-tooling group with 1 update: [pytest](https://github.com/pytest-dev/pytest). Updates `pytest` from 9.1.0 to 9.1.1 - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@9.1.0...9.1.1) --- updated-dependencies: - dependency-name: pytest dependency-version: 9.1.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: benchmark-tooling ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [sphinx-rtd-theme](https://github.com/readthedocs/sphinx_rtd_theme) from 2.0.0 to 3.1.0. - [Changelog](https://github.com/readthedocs/sphinx_rtd_theme/blob/master/docs/changelog.rst) - [Commits](readthedocs/sphinx_rtd_theme@2.0.0...3.1.0) --- updated-dependencies: - dependency-name: sphinx-rtd-theme dependency-version: 3.1.0 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
PyPSA#783) * fix(constraints): freeze empty constraint groups without reshape error CSRConstraint.from_mutable reshaped con.vars with an inferred -1 dimension. For an empty constraint group (zero rows and zero terms) the vars array has size 0, and NumPy refuses to infer a (0, -1) reshape, raising "cannot reshape array of size 0 into shape (0,newaxis)". This broke the documented lossless freeze round-trip for legitimately empty groups (e.g. shifted-time difference constraints at n_time == 1). Pass the explicit _term count instead of -1; NumPy accepts a (0, 0) reshape and the rest of the method already handles zero-row input. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: code review --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…urobi/Xpress/Mosek) (PyPSA#718) * feat(constraints): add _coef_dirty flag + rhs setter short-circuit Tracks per-Constraint coefficient mutation via a single boolean slot, flipped in coeffs/vars/lhs setters. Pure-constant rhs writes now short-circuit and leave coeffs/vars buffers untouched (by identity), so rhs-only updates don't trigger expensive coefficient recompare on the persistent-solver fast path. * feat(persistent): add ModelSnapshot, CoefPattern, StructuralKey Pure-Python snapshot primitives for the persistent-solver Phase 1. Deep-copies value-side fields (var_lb/ub, con_rhs/sign, obj_linear), holds vlabels/clabels by reference, stores canonical CSR (indptr, indices) per constraint container. No Solver import. * feat(persistent): add ModelDiff and compute_diff Pure-function diff for the persistent-solver Phase 1. Detects structural, coord, sparsity, quadratic-objective, value-only var/con, and objective-linear/sense changes. Supports same_model fast path via _coef_dirty and cross-model full re-scan. Includes a focused test suite covering capture, mutation paths, deep-copy invariant, and the same_model toggle. * feat(solvers): add persistent-update orchestration to Solver - supports_persistent_update class flag (default False) - snapshot/_rebuilds/_in_place_updates/_last_rebuild_reason fields - snapshot capture at end of direct _build, _clear_coef_dirty helper - apply_update stub raising UnsupportedUpdate - solve(model, assign) dispatcher with diff-or-rebuild path - update(model, apply=True) primitive returning ModelDiff - threading.Lock around diff+apply+resnapshot - __getstate__/__setstate__ drop native handle and snapshot * test(persistent): smoke test Solver orchestrator with fake backend * feat(solvers): short-circuit rebuild when backend lacks persistent-update support Skip diff computation entirely when supports_persistent_update is False on apply, per plan: 'dispatcher checks flag before calling — if False, skips diffing entirely and goes to rebuild.' * feat(solvers): Gurobi apply_update for persistent solves * feat(solvers): HiGHS apply_update for persistent solves * test(persistent): cross-model, pickle, threading, failure-path coverage * refactor(persistent): row-block numpy snapshot/diff Replace xarray-based snapshot and CSR pattern compare with per-row canonicalised numpy buffers; new ContainerVarUpdate / ContainerRowUpdate payloads. Gurobi/HiGHS apply_update rewritten around batched setAttr / changeColsBounds / changeColsCost / changeColsIntegrality; coefficient writes touch only changed cells. Cross-model diff now ~matches same-model cost for bound/rhs/coef-value sweeps. * feat(persistent): opt-in coord-equality via ignore_dims compute_diff/Solver.solve/Solver.update grow an ignore_dims kwarg. None (default) keeps the current no-coord-check behaviour; any iterable opts into per-container coord-equality on every dim not in the set, supporting rolling-horizon workflows where e.g. the snapshot dim is expected to drift. * feat(persistent): lazy-build Solver, ModelDiff constructors, disallow_rebuild - Solver.from_name now accepts model=None; the first solve(m, ...) builds. - compute_diff folded into ModelDiff.from_snapshot classmethod; new ModelDiff.from_models diffs two linopy models directly. - Solver.solve grows disallow_rebuild=True, which raises RebuildRequiredError instead of falling back to a rebuild. * feat(persistent): opt-in update tracking, snapshot-free ModelDiff.from_models - Add `track_updates` flag (default False) to Solver; skip ModelSnapshot capture when disabled. Raise UpdatesDisabledError on solve(model)/update() if a built solver was constructed without tracking. - Rewrite ModelDiff.from_models to build directly from two models without capturing snapshots; share helpers with from_snapshot. - Update persistent tests to opt into track_updates=True; add coverage for the disabled path. * feat(persistent): wire ModelDiff.from_models for track_updates=False Cross-instance resolves now diff via from_models against the previously built model, with no snapshot. Same-instance mutation still raises UpdatesDisabledError. Snapshot recapture is skipped in this mode. Add cross-instance solve/update tests for the no-snapshot path. * refactor(persistent): cleanups, VarKind enum, fold _clear_coef_dirty Collapse _diff_objective QUAD_OBJ branches; cache n_coef_updates; short-circuit _canonicalize_rows when rows already sorted; tighten buffer extraction. Introduce VarKind enum used across snapshot/diff and HiGHS/Gurobi apply_update; reuse linopy.constants sign tokens. Move _clear_coef_dirty into ModelSnapshot.capture. * refactor(persistent): CSR-backed ContainerConBuffers Source con buffers from Constraint.to_matrix_with_rhs, replacing the dense (n_rows, max_n_term) arrays with CSR (indptr, indices, data). Sign dtype adopts 'U1' across the persistent layer and apply_update in HiGHS/Gurobi consumes CSR-slice payloads instead of -1 masks. Deletes _canonicalize_rows and the _INT64_MAX sentinel. * refactor(persistent): flat-native ModelDiff storage Replace per-container ContainerVarUpdate/ContainerRowUpdate dicts with flat arrays (var_bounds_*, var_type_*, con_coef_* COO, con_rhs_*, con_sign_*) plus VarSlice/ConSlice per-container offsets for diagnostics. Add con_rhs_as_bounds() for ranged-row solvers. Backend apply_update bodies collapse to flat-array calls; remove duplicated label->position resolution. * feat(persistent): apply_update for Xpress and Mosek Implement in-place model updates for Xpress (chgbounds/chgrhs/chgmcoef/ chgrowtype/chgobj/chgobjsense/chgcoltype) and Mosek (chgvarbound/ chgconbound/putaijlist/putclist/putvartypelist/putobjsense). Mosek rejects constraint sign change to trigger rebuild. Consolidate gurobi/highs apply_update tests into a single parametrized file that also covers xpress and mosek. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(persistent): serialize concurrent solves; satisfy mypy * hold solver lock through _run_direct so two threads calling solve(model) on the same Solver no longer race on the native handle (HiGHS returned 0.0 from the second concurrent solve). * narrow Optional ndarrays in persistent.diff.push_var / push_con and in HiGHS/Gurobi/Xpress/Mosek apply_update objective paths. * widen Constraint.rhs setter to ExpressionLike | VariableLike | ConstantLike to match the as_expression call in the body. * widen Constraints.__getitem__(str) return type to Constraint (the dominant case) so tests can set .rhs/.coeffs/.sign without ignores. * add docs for in-place solver updates. * harden coords comparison * Variable.update() / Constraint.update() as canonical mutation API (PyPSA#727) * feat: Variable.update() / Constraint.update() as canonical mutation API Introduces typed ``.update()`` methods on Variable and Constraint as the single, validated, multi-attribute mutation entry point. - ``Variable.update(lower=, upper=)``: validates non-constant inputs are rejected, new dims are rejected, and the resulting ``lower <= upper`` invariant holds across all coords. Returns ``self`` for chaining. - ``Constraint.update(rhs=, sign=)``: constant RHS only. The legacy ``c.rhs = variable`` rearrange-to-lhs path stays on the setter (different semantic, deserves its own explicit method). The existing ``.lower`` / ``.upper`` / ``.sign`` setters become thin shims that forward to ``.update()``, so single-attribute writes (``z.lower = 2``) stay ergonomic and the canonical validation runs in one place. The ``.rhs`` setter forwards constants through ``.update()`` and keeps the expression-rhs rearrange behaviour. This is the on-top experiment for the design discussion on PyPSA#718. ``.lhs`` / ``.coeffs`` / ``.vars`` setters are untouched for now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(update): Constraint.update accepts Variable/Expression rhs Mirrors the existing ``c.rhs = expr`` setter and ``add_constraints`` which both accept mixed-side input and rearrange the residual onto lhs. ``c.update(rhs=x + 5)`` now subtracts ``x`` from lhs and stores ``5`` on rhs. ``.rhs`` setter collapses to a one-line shim. Variable bound rejection of Variable/Expression is kept (bounds are numeric, not symbolic); docstring clarified to spell out that pandas / xarray / numpy arrays are first-class (time-varying bounds). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(update): extend Constraint.update to lhs/coeffs/vars; shim all setters Adds lhs / coeffs / vars to the canonical mutation API. All .lhs / .coeffs / .vars setters now forward to .update() — every Constraint mutation goes through one method with one validation path, one place that flips _coef_dirty. Composition rules: - lhs= replaces the whole expression first; subsequent rhs= rearrangement (Variable/Expression in rhs) sees the new lhs. - lhs= and coeffs= / vars= are mutually exclusive (whole replacement vs partial array update). - sign= is applied last so it composes cleanly. Internal Constraint.sanitize_zeros migrated to update(vars=, coeffs=) — no more internal setter calls in linopy/. 389 tests pass across mutation + persistent-solver suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(update): rename Constraint.update kwarg vars= -> variables= Avoids shadowing Python's vars() builtin. The .vars attribute on Constraint stays (it parallels the .data.vars internal name); only the kwarg gets the unambiguous spelling. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(update): accept positional ConstraintLike in Constraint.update Mirrors add_constraints' dispatch: c.update(x + 5 <= 3) is now shorthand for c.update(lhs=x, sign='<=', rhs=-2), extracted from the AnonymousConstraint / ConstraintBase the comparison produces. Mutually exclusive with the per-attribute kwargs; clear error when mixed. Also reverts the internal sanitize_zeros migration. The setters are pure shims forwarding to update(), so the migration didn't change behaviour or cost — just spelling. The original setter syntax reads more naturally there. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(update): note kwarg form is the targeted, cheap path The positional ConstraintLike form (c.update(x + 5 <= 3)) always rewrites lhs / sign / rhs and flips _coef_dirty. For hot loops that only touch one part, kwarg form (c.update(rhs=...)) skips the unchanged attributes and is materially cheaper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(persistent): default ModelDiff.from_snapshot(same_model=False) Closes the A1 residual from the PyPSA#718 review. The flag-trust path (`skip_coef_compare = same_model and not coef_dirty`) is correct through Constraint.update() (set in one place, shims forward), but `c.coeffs.values[...] = ...` still bypasses _coef_dirty. With same_model=True as the default, that bypass silently produces wrong diffs. Flip the default to False. Cross-model paths (the only production caller, Solver._update_locked, passes explicitly) are unaffected. Same-model warm-update paths now value-diff the CSR data — small perf hit (50-200ms at Mayk-scale per Mayk's bench), correct by default. Solver-aware callers who own the mutation contract can opt back into the optimization with `same_model=True`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: teach .update() in tutorials; mark setters as syntactic sugar - examples/manipulating-models.ipynb: rewrite mutation cells to use Variable.update / Constraint.update; setter form is mentioned in notes as syntactic sugar for the same call. - examples/creating-constraints.ipynb: reframe the CSRConstraint vs Constraint API table around .update() as the mutation API; setters are sugar. - Setter docstrings now say 'syntactic sugar for Constraint/Variable .update; do not add logic here so the contract stays single-sourced' — a directive to future contributors as much as to readers. No deprecation, no breaking change. .update() is the documented canonical mutation API; the seven setters continue to exist as one-line shims. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deprecate(update): warn on mutation setters; promote .update() in docs Adds DeprecationWarning to all seven mutation setters (Variable.lower, Variable.upper, Constraint.coeffs, Constraint.vars, Constraint.sign, Constraint.rhs, Constraint.lhs). Each setter still forwards to .update() so existing code keeps working; the warning points at the canonical API. Internal sanitize_zeros migrated off setters (the last linopy/ caller). api.rst gains Modification sections listing .update() for both Variable and Constraint; tutorial notes rewritten to teach .update() and flag setters as deprecated. Release note added. dual.setter / solution.setter untouched — result assignment, not mutation, different deprecation track. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(update): edge-case coverage; document rhs-rearrangement invariant Constraint.update tests: lhs-only, coeffs-only (vars preserved), compound lhs+sign, mutually-exclusive lhs+coeffs and lhs+variables. Variable.update tests: upper-only, valid array bound. Migrate test_constraint_coef_dirty.py from the now-deprecated setters to .update(), exercising the canonical path; add positional-form and no-op cases. Net effect: same dirty-flag invariants, 7 fewer warnings per pytest run. Docs: Constraint.update rhs= gains a worked example showing the two forms (constant vs variable/expression). add_constraints rhs gets a matching note pointing at the linopy invariant so the rearrangement rule is documented at the creation site too. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review(update): address inline feedback on PyPSA#727 - Constraint._assign_lhs_expr → _assign_lhs (drop redundant suffix; the method already takes a LinearExpression, so the type was in the signature, not the name). - Add Constraint._assign_data(**fields) helper. Wraps the four ``self._data = assign_multiindex_safe(self.data, **kw)`` callsites inside update() (rhs / coeffs / vars / sign). Untouched: the same pattern in dual.setter, sanitize_missings, sanitize_infinities — those aren't update() and stay out of scope here. - Add types.CONSTANT_TYPES tuple, derived from ConstantLike via get_args so the two cannot drift. Variable.update bound validation flipped from negative (isinstance against a hand-rolled non_constant tuple) to positive (isinstance against CONSTANT_TYPES); drops a redundant in-function ``from linopy import expressions`` (the module-level import already covered it). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deprecate(update): FutureWarning on DataArray as variables=; extract _validate_update Constraint - sanitize_zeros now writes _data via _assign_data directly (no longer round-trips through update(variables=DataArray), which would self-trigger the new deprecation warning). - Constraint.update(variables=...) emits FutureWarning when passed a raw DataArray of integer labels; Variable is the supported input. The path stays accepted for back-compat and will be removed alongside the v1 cleanup. Variable - Extract Variable._validate_update(*, lower, upper) — validates, broadcasts, and runs the cross-bound (lb<=ub) check, returning a dict ready for assignment. update() body shrinks to ~3 lines. Coord validation (parity with add_variables) deferred to PyPSA#726 land. Tests - test_constraint_coef_dirty's variables= test now passes a Variable instead of a DataArray (matches the supported input). - test_constraint_vars_setter_with_array wrapped in pytest.warns(FutureWarning) — locks in the deprecation contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(constraints): _assign_data → _update_data; manage _coef_dirty inside The helper now writes fields AND flips _coef_dirty when the written set includes coeffs or vars. Callsites in update() (coeffs / vars branches) and sanitize_zeros drop their explicit `self._coef_dirty = True` lines — the rule lives in one place, can't be forgotten by future field additions. rhs / sign writes still don't dirty (correctly). _assign_lhs is untouched: it uses a different write mechanic (drop_vars + assign) and manages its own flag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * revert(update): rename Constraint.update kwarg variables= back to vars= Restores symmetry with Constraint.vars (property) and data.vars (underlying xarray Dataset key) — the original rename traded one small asymmetry (read vs. mutate kwarg) for a worse one (Python property name vs. Dataset key name). The `vars()` builtin shadowing inside the kwarg position is benign: we never call `vars()` here, and dropping the rename also lets the top-level `linopy.variables` module be used directly inside the function body instead of importing `Variable as _Variable` to dodge the kwarg shadow. Closes PyPSA#730. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * restore(update): keep Constraint.update kwarg as variables= Reverts cf845e1. Under the A/B framing for the .vars naming question, update(variables=...) is Category A — it accepts a linopy.Variable. The previous "restore symmetry with .vars / data.vars" argument conflated two layers: - Public Python API speaks about linopy variables → variables= - Internal xarray Dataset key stays as "vars" (xarray collision on Dataset.variables blocks renaming the key) The asymmetry between property/kwarg name and Dataset key name is principled (API layer vs. storage layer), not arbitrary — same pattern ORMs / serializers use. Keeping variables= here lines up with the broader .vars → .variables direction now being considered for properties. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: ensure dims in solution assignment * refactor(persistent): address review round ModelDiff.from_snapshot/from_models return RebuildReason on rebuild (NONE dropped); diff walkers moved onto _DiffBuilder with context in __init__, single _cat helper. Snapshot buffers share constraint arrays (identity fast path); CSRConstraint.sanitize_zeros copy-on-write. Use isinstance(val, ConstantLike) in Variable._validate_update. * fix: CI failures after master merge Variable.fix()/unfix() set both bounds atomically via update() instead of sequential deprecated setters (tripped new lower<=upper cross-validation). Fail fast on quadratic lhs in Constraint.update; type-narrowing fixes for mypy. * refactor(persistent): lazy COO expansion of coefficient diffs ModelDiff stores per-container _CoefDelta (changed rows referencing the CSR buffers); con_coef_rows/cols/vals materialize on first access via cached property. Expansion is now vectorized; backends guard on n_coef_updates. Follows coroa's suggestion on PR 718. * refactor(persistent): assemble snapshot from diff walk; pure capture (A2+A6) _DiffBuilder records target buffers/coords; ModelDiff.snapshot replaces the O(nnz) re-capture after in-place updates. ModelSnapshot.capture no longer mutates the model: the _coef_dirty clear moves to the solver, coupled to snapshot adoption (build + successful apply, never on apply=False). * refactor(persistent): template-method apply_update (A4, D2-D4) Base Solver orchestrates the diff sections and validates up front (sign support; Mosek semi-continuous now fails before any native mutation); backends implement _apply_* hooks. Binary [0,1] re-clamp lifted to base with Gurobi no-op (VType 'B' implies bounds natively). self.sense now set uniformly; HiGHS vtype map cached; Xpress/Mosek list-conversion helpers. * refactor(persistent): lock-free diff preview, document solve atomicity (A5) update(model, apply=False) computes the diff without the solver lock (immutable snapshot buffers, same_model=False since _coef_dirty cannot be trusted concurrently). solve keeps the coarse lock: apply->run must be atomic and native handles are not thread-safe. Tests pin the non-blocking preview and the preview/apply asymmetry for raw .values mutations. * refactor(persistent): split diff/apply, namedtuple ctx (coroa review) Replace _update_locked(apply=...) with _compute_diff + _apply_locked, dropping the dead apply=False branches. Read supports_* off the instance and give Gurobi's apply context a _GurobiApplyCtx namedtuple. * chore(codespell): whitelist 'coo' (sparse COO format) to fix false positive * refactor(persistent): delegate from_models to from_snapshot ModelDiff.from_models now captures a snapshot of model_a and defers to from_snapshot(same_model=False), removing the duplicated baseline-extraction and diff loops. * refactor(persistent): drop dead per-container structural guards The global _structural_reason precheck (via vlabels/clabels, the concatenation of per-container active_labels) already pins each container's active_labels and shape before diff_var/diff_con run, so the per-container shape/active_labels mismatch branches were unreachable duplication that silently degraded to a rebuild. Removed from diff_var, diff_con, and diff_objective. * test(persistent): cover diff rebuild reasons and snapshot var kinds Adds coverage for QUAD_OBJ, variable-type change, sign-only mutation, top-level STRUCTURAL_LABELS (vlabels/clabels), indices SPARSITY, binary/ integer/semi-continuous capture, and ModelDiff inspect/repr helpers. persistent/diff.py 81->96%, snapshot.py 87->99%. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Felix <117816358+FBumann@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bumps [numpydoc](https://github.com/numpy/numpydoc) from 1.7.0 to 1.10.0. - [Release notes](https://github.com/numpy/numpydoc/releases) - [Changelog](https://github.com/numpy/numpydoc/blob/main/RELEASE.rst) - [Commits](numpy/numpydoc@v1.7.0...v1.10.0) --- updated-dependencies: - dependency-name: numpydoc dependency-version: 1.10.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
* fix(variables): Variable.where works with solution attached Variable.where passed a fill dict covering only labels/lower/upper to Dataset.where, which requires an exact match against the dataset's data variables. After solve() or fix() the variable carries extra columns (solution, stashed bounds), so where() raised a ValueError. Augment the fill dict to cover all present data variables, defaulting extras to NaN. * fix(variables): annotate where fill dict for mypy
* fix(highs): pass uint8 integrality array to changeColsIntegrality highspy's changeColsIntegrality expects var_types as a uint8 array (matching HiGHS's uint8-backed HighsVarType enum). The persistent update path already builds it as uint8 (_apply_var_types); align the direct-solve path so it matches highspy's tightened type stubs and is robust across versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(highs): silence assignment error on highspy lazy module highspy is imported under TYPE_CHECKING, so mypy types `highspy` as the module; assigning the `_LazyModule` shim then trips [assignment] once highspy ships recognized types (highspy 1.13.1). Mirror the existing gurobipy line with `# type: ignore[assignment]`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use the @AGENTS.md import so Claude Code loads the contribution rules, workflow and conventions automatically instead of relying on the agent to follow the link. Keeps AGENTS.md as the single source of truth. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
…PSA#797, PyPSA#796) (PyPSA#798) * feat(piecewise): partial `active` gate support via active_gate helper (PyPSA#796) `add_piecewise_formulation(active=...)` assumed the gate covered the whole indexed dimension. A gate defined over only a subset of coordinate labels (or with masked entries) silently forced the uncovered entries to zero — breaking mixed committable / non-committable formulations (PyPSA#1755). - Add `linopy.active_gate(active, coords, fill_value=1)`: pads a partial gate to full coverage, treating missing/masked entries as always-active (1) or always-off (0). Lives in its own module `linopy/_active_gate.py` as a temporary legacy stopgap; under v1 the bare `active.reindex(coords).fillna(fill_value)` idiom suffices and the helper is expected to be deprecated. - `add_piecewise_formulation` now rejects an under-defined `active` (strict subset or masked) with an actionable error instead of mis-solving. A lower-dimensional gate still broadcasts and is accepted. - Docs (api, piecewise guide, release notes) and tests in a dedicated `test/test_piecewise_active_gate.py` (parametrized over both partial shapes and incremental/sos2/disjunctive paths). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(piecewise): use LinearExpression.has_terms for gate gap detection Apply review suggestion (PyPSA#797): replace the hand-rolled vars/term-dim reduction with the public `has_terms` property in `active_gate` and the coverage validation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(piecewise): replace active_gate helper with active_fill parameter Per PR PyPSA#797 review: instead of a standalone `active_gate` helper, gate a partial `active` via an `active_fill` parameter on `add_piecewise_formulation`. The function derives its own coordinate, so the caller supplies nothing extra; `active_fill=None` (default) keeps the function strict (partial `active` raises), and `0`/`1` opt into always-off / always-on. - Drop `linopy/_active_gate.py` and the `active_gate` export. - Add `active_fill: int | None` to `add_piecewise_formulation`; fold the guard + padding into `_resolve_active`. - `active_fill` is transitional (removed once v1 makes `active.reindex(coords).fillna(value)` sufficient) — documented as such. - Tests renamed to test_piecewise_active_fill.py; docs updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(piecewise): state explicitly that a partial active = subset labels or masked Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(piecewise): type active_fill as int to satisfy mypy Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # doc/release_notes.rst # linopy/common.py # linopy/config.py # linopy/matrices.py # linopy/model.py # linopy/variables.py # test/test_constraints.py
CSRConstraint._to_dataset hardcoded np.int64 for the reconstructed vars and labels arrays, so a frozen constraint round-tripped through .data/.flat came back as int64 regardless of options["label_dtype"], silently undoing the int32 memory win. Use options["label_dtype"] to match the allocation paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes proposed in this Pull Request
Cut memory for internal integer arrays (labels, vars indices,
_termcoords) by ~20% and improve build speed on large models by defaulting toint32instead ofint64.What changed
linopy/constants.py: AddedDEFAULT_LABEL_DTYPE = np.int32linopy/model.py: Variable and constraint label assignment usesnp.arange(..., dtype=DEFAULT_LABEL_DTYPE)with overflow guard that raisesValueErrorif labels exceed int32 max (~2.1 billion)linopy/expressions.py:_termcoord assignment and.astype(int)for vars arrays now useDEFAULT_LABEL_DTYPElinopy/common.py:fill_missing_coordsuses int32 arange; polars schema infersInt32/Int64based on actual array dtype instead of OS/numpy-version heuristictest/test_constraints.py: Updated dtype assertions to usenp.issubdtype(compatible with both int32 and int64)test/test_dtypes.py(new): Tests for int32 labels, expression vars, solve correctness, and overflow guarddev-scripts/benchmark_lp_writer.py(new): Benchmark script supporting--phase memory|build|lp_writewith--plotcomparison modeBenchmark results
Reproduce with:
Memory (dataset
.nbytes)Consistent 1.25x reduction across all problem sizes (e.g. 640 MB → 512 MB at 8M vars). The
labelsandvarsarrays shrink 50% (int64 → int32) whilelower/upper/coeffs/rhsstay float64.Build speed
No regression on small/medium models. ~2x speedup at largest sizes (4.5M–8M vars) due to reduced memory pressure.
Similar results on real pypsa model
Checklist
doc.doc/release_notes.rstof the upcoming release is included.