From e2cad328349f7430d29c611384538c19016ba8fd Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 29 Jun 2026 19:14:06 +0200 Subject: [PATCH 01/33] test: MultiIndex -> flat+aux feasibility checks (#744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prove a first-class pd.MultiIndex snapshot can be replaced by a flat dim + level aux coords without changing the model, by explicit equality checks: reset_index normalization, where/groupby selection, per-period roll (#751), group-by-level (#751), an equivalent solved LP, and the output re-stack round-trip (flat solution -> MI via the caller's own mapping). Pins that Variable.sel does not forward MI tuple-select (#752 §2). Both semantics. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 130 +++++++++++++++ test/test_mi_feasibility.py | 160 +++++++++++++++++++ 2 files changed, 290 insertions(+) create mode 100644 arithmetics-design/multiindex-feasibility.md create mode 100644 test/test_mi_feasibility.py diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md new file mode 100644 index 00000000..71ddddbb --- /dev/null +++ b/arithmetics-design/multiindex-feasibility.md @@ -0,0 +1,130 @@ +# MultiIndex feasibility for v1 (#744) + +> Verification note (Claude Code, prompted by @FBumann), 2026-06-29. Tracks +> whether linopy can drop **first-class `pd.MultiIndex`** support in v1 in favour +> of a **flat dim + auxiliary level coords** model. Discussion home: +> [#744](https://github.com/PyPSA/linopy/issues/744). Equality checks are tracked +> in [`test/test_mi_feasibility.py`](../test/test_mi_feasibility.py) (runs under +> both `legacy` and `v1`). + +**Question.** Can v1 drop the stacked `pd.MultiIndex` snapshot for a flat +`snapshot` dim carrying `period`/`timestep` as auxiliary level coords? + +**"Feasible"** has a precise, testable meaning: every real MI use case has a +flat+aux form that builds an *equivalent model* — proven by an explicit equality +check, not asserted. + +## Data model under test + +| | today (C) | proposed (flat+aux) | +|---|---|---| +| snapshot dim | `MultiIndex[(period, timestep)]` | flat `snapshot` dim | +| level identity | MI levels | `period`/`timestep` **aux coords** on `snapshot` | +| entry conversion | — | `obj.reset_index("snapshot")` (canonical xarray, no custom logic) | +| alignment | tuple-identity | **positional** (one canonical snapshot order) | + +The entry conversion is byte-identical across linopy's supported xarray range +(2024.2.0 floor → 2026.4.0); it post-dates xarray's explicit-indexes refactor +(~2022.06, below the floor), so no compat shim is needed. + +## Feasibility matrix + +Two axes, deliberately separate: +- **feasible** — can flat+aux build an *equivalent model*? ✅ verified (has a test) · ☐ open. +- **desirable** — is the flat+aux form at least as good, across three facets — *nicer* (ergonomics), *safer* (catches errors), *flexible* (coords stay manipulable)? 👍 better · ➖ parity · ⚠️ friction/cost · ☐ TBD. + +The `entry`–`model` rows are tracked in `test/test_mi_feasibility.py`; PyPSA links +are pinned at **v1.2.4** (commit [`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). + +| op | MI form | flat+aux form | feasible | desirable | PyPSA call site @ v1.2.4 | +|---|---|---|---|---|---| +| **entry** | `coords=[mi]` | `reset_index(dim)` | ✅ | 👍 deletes MI machinery | — | +| **select** | `sel(snapshot=(p, slice))` | `where(period == p)` | ✅ | ➖ parity | [`constraints.py` L1235‑1248](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1235-L1248) (KVL per-period) | +| **roll** | per-period `sel`-loop | `groupby("period").roll` | ✅ (#751) | 👍 deletes `.data` loop | [`constraints.py` L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694) (SOC) | +| **group** | level groupby | `groupby("period").sum()` | ✅ (#751) | 👍 flat works (#751); MI level-groupby broken upstream ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | — | +| **model** | MI per-period LP | flat+aux LP (solved `==`) | ✅ | ➖ parity | — | +| **soc** | `.data.sel().roll` + `FILL_VALUE` rebuild | previous-SOC via `groupby("period").roll` | ☐ | 👍 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908) | +| **stoch** | stochastic = 2nd MI (scenario) | two aux-coord groups (`period`+`scenario`) | ☐ | ☐ TBD | — | +| **output** | `solution`/`dual` MI-indexed | flat + level coords (A) or re-stacked (B) | ☐ | ⚠️ A/B trade-off | — | +| **n.snapshots** | `pd.MultiIndex` public API | flat dim + level coords | ☐ | ⚠️ PyPSA API migration | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | + +The per-op cells above are the *nicer* facet. Two more facets are +**representation-wide**, not per-op (both verified this session): + +- **safer** — under v1 a conflicting level/aux coord *raises* (the general + aux-coord-conflict rule, `linopy/semantics.py: enforce_aux_conflict`); legacy + silently keeps one side ([#295](https://github.com/PyPSA/linopy/issues/295)). + MI "avoids" the conflict only by locking the levels into the index. +- **flexible** — flat+aux level coords can be `drop_vars`/`rename`/`assign_coords`'d + like any coord; on an MI the same reassignment raises *"cannot drop or update + coordinate … would corrupt the index"*. Removing MI removes that rigidity. + +The two axes tell different stories: flat+aux is **feasible almost everywhere**, +and **desirable for every build-time op** — nicer (deletes MI machinery and +PyPSA's `.data`/`FILL_VALUE` workarounds), safer, and more flexible. The only ⚠️ +cost sits at the **public boundary** (`output`, `n.snapshots`), which is exactly +the A-vs-B question. So "is it nice?" and "is it possible?" point the same way for +the internals; the debate is purely about the user-facing API. + +**Side finding (not a row):** `Variable.sel` can't MI-tuple-select +(`x.sel(snapshot=(p, slice))` → `InvalidIndexError`), which is why PyPSA drops to +`.data` ([#752](https://github.com/PyPSA/linopy/issues/752) §2). Under flat+aux it +becomes `where(period == p)` / `isel`, removing the internals reach. Pinned by +`test_variable_mi_tuple_sel_not_forwarded`. + +The `entry`–`model` rows answer the **steady-state (linopy)** question; the open +rows are **PyPSA-owned** and answer the **transition** question — verified by +solution-equivalence on real networks (multi-period, stochastic, Monte-Carlo) +plus scoping the public `n.snapshots` change. + +## Sub-decision: the snapshot dim coordinate + +After `reset_index`, the dim is **coordinate-less** (xarray virtualizes `0..N-1` +on access). `timestep` can't be the coord (non-unique across periods); the unique +`(period, timestep)` pair *is* an MI. So: + +| dim coordinate | alignment | `.sel(snapshot=int)` | `.sel` by level tuple | +|---|---|---|---| +| pure `reset_index` (none, virtual `0..N-1`) | positional | ✅ positional fallback | ❌ → `where(period==…)` | +| `+ assign_coords(RangeIndex)` (stored int index) | by label (`==` position here) | ✅ label lookup | ❌ → `where(period==…)` | + +Integer `.sel(snapshot=k)` works **either way** — on the coordinate-less dim +xarray degrades to positional selection (the tell: out-of-range raises +`IndexError`, not a label `KeyError`); with a `RangeIndex` it's a label lookup. +Since the coord is `0..N-1`, label `==` position, so they're indistinguishable for +valid integers. What flat+aux genuinely loses is `.sel(snapshot=(2020, "t1"))` — +selection by the **level tuple** (needs the MI) — replaced by `where(period==…)`; +that capability is gone under *both* flat variants, so it is not what the +coordinate choice decides. + +The only thing the coordinate choice decides is **alignment**: positional +(coordinate-less) vs label (`RangeIndex`). For snapshots this is a distinction +without a difference — one canonical `n.snapshots` order means positional and +label alignment coincide — and it matches the model linopy already uses for a +plain single-period datetime snapshot dim. The single line for §11 is just: +**snapshot alignment is positional, not tuple-identity.** + +## Transition shape (PyPSA) + +PyPSA's hard MI couplings are a short, enumerated set +([#752](https://github.com/PyPSA/linopy/issues/752)) — the SOC roll + `FILL_VALUE` +rebuild and the growth `reindex_like(lhs.data)` linked above. These are *reluctant +workarounds*: the code comment says *"internal xarray multi-index difficulties"* +(cf. pydata/xarray#6836). #751 fixed the level groupby, so migrating these sites +**deletes** the `.data`-reach rather than re-spelling it. The build-time sites +break under **A and B equally** (they `.sel` the *stored* MI mid-build), so they +migrate regardless; A vs B is decided only by the **public `n.snapshots`** question +(`output`/`n.snapshots` rows), pressure-tested against +[PyPSA#1484](https://github.com/PyPSA/PyPSA/issues/1484) (Monte-Carlo, which wants +an MI level per sampled dimension). + +## Decision record (to fill once the open rows close) + +> _Outcome, the chosen option (A — return flat / B — re-stack MI at the boundary), +> the `n.snapshots` migration scope, and the evidence rows that justify it. This +> note, once complete, is what closes #744._ + +- **Internal normalization to flat+aux:** _safe to adopt regardless of A/B_ — + cheap, canonical, version-safe (`entry` row + version sweep). **[provisional]** +- **A vs B:** _TBD_ — gated on the `output`/`n.snapshots` rows and the PyPSA + stochastic experiments. diff --git a/test/test_mi_feasibility.py b/test/test_mi_feasibility.py new file mode 100644 index 00000000..4cb42986 --- /dev/null +++ b/test/test_mi_feasibility.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +""" +MultiIndex -> flat dim + aux coords: feasibility for v1 (#744). + +Can v1 drop first-class ``pd.MultiIndex`` snapshots for a flat ``snapshot`` dim +with ``period``/``timestep`` as auxiliary level coords? Each test below is an +explicit equality check (MI form vs flat+aux form), run under both legacy and v1 +semantics. + +The matrix, findings, and pinned PyPSA call sites live in the discussion +artifact: ``arithmetics-design/multiindex-feasibility.md``. +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd +import pytest +import xarray as xr + +from linopy import Model, available_solvers +from linopy.testing import assert_linequal + +PERIODS = [2020, 2030] +N = 6 # 2 periods x 3 timesteps +PERIOD_OF = np.repeat(PERIODS, 3) # period per flat snapshot +STEP_OF = np.tile(["t1", "t2", "t3"], 2) +DEMAND = {2020: 5.0, 2030: 7.0} + +needs_highs = pytest.mark.skipif( + "highs" not in available_solvers, reason="highs solver not available" +) + + +def _mi() -> pd.MultiIndex: + return pd.MultiIndex.from_product( + [PERIODS, ["t1", "t2", "t3"]], names=["period", "timestep"] + ) + + +def _flat() -> dict: + """Flat snapshot dim with period/timestep as aux coords.""" + s = pd.RangeIndex(N, name="snapshot") + return { + "snapshot": s, + "period": xr.DataArray(PERIOD_OF, dims="snapshot", coords={"snapshot": s}), + "timestep": xr.DataArray(STEP_OF, dims="snapshot", coords={"snapshot": s}), + } + + +# --- data-model equivalence (xarray level) ---------------------------------- # +def test_entry_normalize() -> None: + """reset_index *is* the conversion: levels -> aux coords, dim coordinate-less.""" + r = xr.DataArray( + np.arange(N), coords={"snapshot": _mi()}, dims="snapshot" + ).reset_index("snapshot") + assert list(r.coords) == ["period", "timestep"] + assert "snapshot" not in r.indexes # coordinate-less; xarray virtualizes 0..N-1 + assert np.array_equal(r["snapshot"].values, np.arange(N)) + + +def test_level_selection() -> None: + """where(period == p) reproduces sel(snapshot=(p, slice)).""" + mi = xr.DataArray(np.arange(N), coords={"snapshot": _mi()}, dims="snapshot") + flat = xr.DataArray(np.arange(N), coords=_flat(), dims="snapshot") + for p in PERIODS: + assert np.array_equal( + mi.sel(snapshot=(p, slice(None))).values, + flat.where(flat.period == p, drop=True).values, + ) + + +def test_per_period_roll() -> None: + """PyPSA SOC pattern: groupby('period').roll == per-period sel-loop (needs #751).""" + mi = xr.DataArray(np.arange(N), coords={"snapshot": _mi()}, dims="snapshot") + flat = xr.DataArray(np.arange(N), coords=_flat(), dims="snapshot") + mi_rolled = np.concatenate( + [mi.sel(snapshot=(p, slice(None))).roll(snapshot=1).values for p in PERIODS] + ) + flat_rolled = flat.groupby("period").map(lambda b: b.roll(snapshot=1)).values + assert np.array_equal(mi_rolled, flat_rolled) + + +def test_groupby_level_name() -> None: + """Group a LinearExpression by a level name == the slow fallback (#751).""" + m = Model() + x = m.add_variables(coords=[pd.RangeIndex(N, name="snapshot")], name="x") + expr = (1.0 * x).assign_coords(period=xr.DataArray(PERIOD_OF, dims="snapshot")) + assert_linequal( + expr.groupby("period").sum(), expr.groupby("period").sum(use_fallback=True) + ) + + +def test_variable_mi_tuple_sel_not_forwarded() -> None: + """Logged finding: Variable.sel can't MI-tuple-select -> PyPSA drops to .data (#752 §2).""" + m = Model() + x = m.add_variables(coords=[pd.Index(_mi(), name="snapshot")], name="x") + with pytest.raises(pd.errors.InvalidIndexError): + x.sel(snapshot=(2020, slice(None))) + + +# --- model equivalence (linopy level, solved) ------------------------------- # +def _solve(snapshot, add_demand) -> tuple[float, np.ndarray]: + """min-cost x s.t. per-period demand; constraints built by ``add_demand``.""" + m = Model() + x = m.add_variables(lower=0, upper=4.0, coords=[snapshot], name="x") + add_demand(m, x) + m.add_objective((np.arange(1.0, N + 1.0) * x).sum()) + m.solve(solver_name="highs", output_flag=False) + return float(m.objective.value), np.sort(m.solution["x"].values) + + +@needs_highs +def test_per_period_lp_equivalent() -> None: + """Same per-period-demand LP, MI vs flat+aux -> identical optimum.""" + + def mi_demand(m, x): # select by position -- MI tuple-sel is not forwarded + for p, d in DEMAND.items(): + m.add_constraints( + x.isel(snapshot=np.flatnonzero(PERIOD_OF == p)).sum() >= d + ) + + def flat_demand(m, x): # the level rides as an aux coord -> groupby + e = (1.0 * x).assign_coords(period=_flat()["period"]) + rhs = xr.DataArray( + list(DEMAND.values()), dims="period", coords={"period": PERIODS} + ) + m.add_constraints(e.groupby("period").sum() >= rhs) + + obj_mi, sol_mi = _solve(pd.Index(_mi(), name="snapshot"), mi_demand) + obj_flat, sol_flat = _solve(_flat()["snapshot"], flat_demand) + assert np.isclose(obj_mi, obj_flat) + assert np.allclose(sol_mi, sol_flat) + + +@needs_highs +def test_output_restacks_to_mi() -> None: + """ + output: a flat solution re-stacks to MI at the boundary, as PyPSA would. + + linopy returns a bare flat ``snapshot`` dim; the caller reconstructs the MI + with the mapping it already owns (``n.snapshots``) -- one ``assign_coords``, + values and order preserved. The inverse of the ``entry`` row. + """ + m = Model() + x = m.add_variables(lower=0, upper=4.0, coords=[_flat()["snapshot"]], name="x") + e = (1.0 * x).assign_coords(period=_flat()["period"]) + rhs = xr.DataArray(list(DEMAND.values()), dims="period", coords={"period": PERIODS}) + m.add_constraints(e.groupby("period").sum() >= rhs) + m.add_objective((np.arange(1.0, N + 1.0) * x).sum()) + m.solve(solver_name="highs", output_flag=False) + + sol = m.solution["x"] # bare flat dim, no level coords carried through solve + coords = xr.Coordinates.from_pandas_multiindex( + _mi(), "snapshot" + ) # explicit-index API + restacked = sol.assign_coords(coords) # caller's own snapshot mapping + assert isinstance(restacked.indexes["snapshot"], pd.MultiIndex) + assert np.array_equal(restacked.values, sol.values) + assert list(restacked.to_dataframe().index.names) == ["period", "timestep"] From ba29401328f457e9b6346e173ee1ae0e8851fdaa Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 29 Jun 2026 19:14:07 +0200 Subject: [PATCH 02/33] docs(arithmetics): MultiIndex -> flat+aux feasibility matrix (#744) Discussion artifact for the v1 MultiIndex decision: feasible (verified) vs desirable (opinion) matrix per op, with PyPSA call sites pinned at v1.2.4; the snapshot dim-coordinate sub-decision; the conclusion that linopy drops MultiIndex from its mental model (flat in, flat out, output re-stack owned by PyPSA); and the open PyPSA-side rows (stochastic 2nd MI, n.snapshots). Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 113 +++++++++++++------ 1 file changed, 76 insertions(+), 37 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 71ddddbb..864bb32c 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -16,7 +16,7 @@ check, not asserted. ## Data model under test -| | today (C) | proposed (flat+aux) | +| | today (MI) | proposed (flat+aux) | |---|---|---| | snapshot dim | `MultiIndex[(period, timestep)]` | flat `snapshot` dim | | level identity | MI levels | `period`/`timestep` **aux coords** on `snapshot` | @@ -29,24 +29,46 @@ The entry conversion is byte-identical across linopy's supported xarray range ## Feasibility matrix -Two axes, deliberately separate: -- **feasible** — can flat+aux build an *equivalent model*? ✅ verified (has a test) · ☐ open. -- **desirable** — is the flat+aux form at least as good, across three facets — *nicer* (ergonomics), *safer* (catches errors), *flexible* (coords stay manipulable)? 👍 better · ➖ parity · ⚠️ friction/cost · ☐ TBD. +Two axes — **feasible** (can flat+aux build an *equivalent model*?) and +**desirable** (our *opinion*: is it at least as good — *nicer* / *safer* / +*flexible*?): -The `entry`–`model` rows are tracked in `test/test_mi_feasibility.py`; PyPSA links +| bubble | feasible | desirable | +|---|---|---| +| 🟢 | works today (tested) | better | +| 🔵 | achievable (unimplemented) | — | +| ⚪ | — | parity | +| 🔴 | not feasible | worse | + +The 🟢 rows are tracked in `test/test_mi_feasibility.py`; PyPSA links are pinned at **v1.2.4** (commit [`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). | op | MI form | flat+aux form | feasible | desirable | PyPSA call site @ v1.2.4 | |---|---|---|---|---|---| -| **entry** | `coords=[mi]` | `reset_index(dim)` | ✅ | 👍 deletes MI machinery | — | -| **select** | `sel(snapshot=(p, slice))` | `where(period == p)` | ✅ | ➖ parity | [`constraints.py` L1235‑1248](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1235-L1248) (KVL per-period) | -| **roll** | per-period `sel`-loop | `groupby("period").roll` | ✅ (#751) | 👍 deletes `.data` loop | [`constraints.py` L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694) (SOC) | -| **group** | level groupby | `groupby("period").sum()` | ✅ (#751) | 👍 flat works (#751); MI level-groupby broken upstream ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | — | -| **model** | MI per-period LP | flat+aux LP (solved `==`) | ✅ | ➖ parity | — | -| **soc** | `.data.sel().roll` + `FILL_VALUE` rebuild | previous-SOC via `groupby("period").roll` | ☐ | 👍 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908) | -| **stoch** | stochastic = 2nd MI (scenario) | two aux-coord groups (`period`+`scenario`) | ☐ | ☐ TBD | — | -| **output** | `solution`/`dual` MI-indexed | flat + level coords (A) or re-stacked (B) | ☐ | ⚠️ A/B trade-off | — | -| **n.snapshots** | `pd.MultiIndex` public API | flat dim + level coords | ☐ | ⚠️ PyPSA API migration | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | +| **entry** | `coords=[mi]` | `reset_index(dim)` | 🟢 | 🟢 deletes MI machinery | [`constraints.py` L1052](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1052) (`from_pandas_multiindex`) | +| **level select** | `sel(snapshot=(p, slice))` | `where(period == p)` | 🟢 | ⚪ parity | [`constraints.py` L1235‑1248](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1235-L1248) (KVL; the per-period loop is topology-driven — `cycle_matrix(period)` — not MI, so flat+aux only swaps the `sns.get_loc` slice for `where`, the loop stays) | +| **period roll** | `roll(1)` + `_period_start_mask` | `groupby("period").map(roll)` | 🟢 (#751) | 🟢 mask-free (boundary from grouping) | [`constraints.py` L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694) (SOC) | +| **level groupby** | `groupby(MI level)` ❌ broken ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | `groupby("period").sum()` | 🟢 (#751) | 🟢 not just *nicer* — MI is *broken*, not merely workaround-y; flat+aux groups by the aux coord and just works (#751) | — | +| **solve LP** | per-period `isel`-sum `≥ d`, solved | `groupby("period").sum() ≥ d`, solved `==` | 🟢 | ⚪ parity | — | +| **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`; `where(period==…)` mask | 🔵 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | +| **stochastic** | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ same mechanism as snapshot (name axis) — but PyPSA is *adding* MI here, not removing it | scenario MI [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; round-trip MI [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | +| **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | +| **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🔵 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | +| **n.snapshots** | `pd.MultiIndex` public API | flat dim + level coords | 🔵 | 🔴 PyPSA API migration | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | + +**No row needs a linopy change to be feasible.** The `entry` conversion is a +user-side `reset_index` with today's linopy; linopy auto-accepting `coords=[mi]` +and decomposing it would be *optional* input sugar. Dropping MI is a simplification +linopy chooses, not a capability it must add. + +**`level groupby` is the one row where flat+aux is *necessary*, not merely +preferable.** Elsewhere flat+aux is parity or deletes a workaround for something MI +still does; here grouping by an MI *level* is broken upstream (xarray#6836, outside +linopy's control) — there is no working native path at all. flat+aux groups by the +aux-coord *name*, which #751 routed onto the existing fast path. (The dense-`_term` +memory cost of groupby-sum, [#756](https://github.com/PyPSA/linopy/issues/756)/[#757](https://github.com/PyPSA/linopy/issues/757), +is real but representation-agnostic — both forms hit the same `unstack` — so it is +*not* what separates MI from flat+aux here.) The per-op cells above are the *nicer* facet. Two more facets are **representation-wide**, not per-op (both verified this session): @@ -59,12 +81,21 @@ The per-op cells above are the *nicer* facet. Two more facets are like any coord; on an MI the same reassignment raises *"cannot drop or update coordinate … would corrupt the index"*. Removing MI removes that rigidity. -The two axes tell different stories: flat+aux is **feasible almost everywhere**, +The two axes tell different stories: flat+aux is **feasible almost everywhere** and **desirable for every build-time op** — nicer (deletes MI machinery and -PyPSA's `.data`/`FILL_VALUE` workarounds), safer, and more flexible. The only ⚠️ -cost sits at the **public boundary** (`output`, `n.snapshots`), which is exactly -the A-vs-B question. So "is it nice?" and "is it possible?" point the same way for -the internals; the debate is purely about the user-facing API. +PyPSA's `.data`/`FILL_VALUE` workarounds), safer, and more flexible. The output +boundary is a **cheap conversion PyPSA owns** (`output` row, tested), and the one MI that lived *inside* a linopy object — `model.parameters.snapshots` (`snapshots param` row) — goes flat too. So the +conclusion for linopy is clean: **accept MI as input sugar, decompose on entry +(`reset_index`), and never reconstruct — flat in, flat out. linopy drops +MultiIndex from its mental model entirely.** + +The remaining cost is **not inside linopy**: it is (a) whether PyPSA keeps +`n.snapshots` as MI — a cheap boundary wrap it can do on its own side, decoupled +from this decision — and (b) the **stochastic / Monte-Carlo direction** +(`stochastic` row), a *second* MI `(scenario, name)` that PyPSA is actively +entrenching ([#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants an MI level +per sampled dimension). That is the genuine open risk, and it is a PyPSA-side +call — linopy works either way. **Side finding (not a row):** `Variable.sel` can't MI-tuple-select (`x.sel(snapshot=(p, slice))` → `InvalidIndexError`), which is why PyPSA drops to @@ -72,7 +103,7 @@ the internals; the debate is purely about the user-facing API. becomes `where(period == p)` / `isel`, removing the internals reach. Pinned by `test_variable_mi_tuple_sel_not_forwarded`. -The `entry`–`model` rows answer the **steady-state (linopy)** question; the open +The 🟢 rows answer the **steady-state (linopy)** question; the 🔵 rows are **PyPSA-owned** and answer the **transition** question — verified by solution-equivalence on real networks (multi-period, stochastic, Monte-Carlo) plus scoping the public `n.snapshots` change. @@ -106,25 +137,33 @@ plain single-period datetime snapshot dim. The single line for §11 is just: ## Transition shape (PyPSA) -PyPSA's hard MI couplings are a short, enumerated set -([#752](https://github.com/PyPSA/linopy/issues/752)) — the SOC roll + `FILL_VALUE` -rebuild and the growth `reindex_like(lhs.data)` linked above. These are *reluctant -workarounds*: the code comment says *"internal xarray multi-index difficulties"* -(cf. pydata/xarray#6836). #751 fixed the level groupby, so migrating these sites -**deletes** the `.data`-reach rather than re-spelling it. The build-time sites -break under **A and B equally** (they `.sel` the *stored* MI mid-build), so they -migrate regardless; A vs B is decided only by the **public `n.snapshots`** question -(`output`/`n.snapshots` rows), pressure-tested against +[#752](https://github.com/PyPSA/linopy/issues/752) catalogues PyPSA **reaching +into linopy internals** (term-storage: `.data`, `vars`/`coeffs`/`const`, `_term`, +the `FILL_VALUE` sentinel) — *not* MI per se. But several of those reaches are +**MI-driven workarounds**: the SOC `.data.sel().roll` + `FILL_VALUE` rebuild and +the growth `reindex_like(lhs.data)` linked above exist because MI groupby is broken +(the code comment says *"internal xarray multi-index difficulties"*, cf. +pydata/xarray#6836). #751 fixed the level groupby, so flat+aux **deletes** those +specific reaches rather than re-spelling them. These build-time sites `.sel` the +*stored* MI mid-build, so they migrate whether or not PyPSA re-stacks the output. +What's left to decide is only the **public `n.snapshots`** question and the +stochastic 2nd MI (`n.snapshots`/`stochastic` rows), pressure-tested against [PyPSA#1484](https://github.com/PyPSA/PyPSA/issues/1484) (Monte-Carlo, which wants an MI level per sampled dimension). ## Decision record (to fill once the open rows close) -> _Outcome, the chosen option (A — return flat / B — re-stack MI at the boundary), -> the `n.snapshots` migration scope, and the evidence rows that justify it. This -> note, once complete, is what closes #744._ - -- **Internal normalization to flat+aux:** _safe to adopt regardless of A/B_ — - cheap, canonical, version-safe (`entry` row + version sweep). **[provisional]** -- **A vs B:** _TBD_ — gated on the `output`/`n.snapshots` rows and the PyPSA - stochastic experiments. +> _The `n.snapshots` scope, the stochastic-MI resolution, and the evidence rows +> that justify them. This note, once complete, is what closes #744._ + +- **linopy drops MultiIndex from its mental model** — accept MI as input sugar, + decompose on entry (`reset_index`), never reconstruct (flat in, flat out). + Internal normalization is safe to adopt now: cheap, canonical, version-safe + (`entry` row + version sweep). **[provisional — internals verified]** +- **Output: linopy returns flat** — the re-stack is a cheap boundary conversion + PyPSA owns (`output` row, tested); no MI adapter lives inside linopy. +- **`n.snapshots`** — PyPSA's independent, decoupled choice: keep MI (wrap at its + own boundary) or flatten. _TBD, PyPSA-side._ +- **Stochastic 2nd MI `(scenario, name)`** (`stochastic` row) — the genuine open risk; + PyPSA is entrenching it ([#1484](https://github.com/PyPSA/PyPSA/issues/1484)). + Resolve via a PyPSA-side solution-equivalence spike. _TBD._ From 743a3afd7c419889221c5e1fcfff7ffe7cc8c040 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 29 Jun 2026 20:56:14 +0200 Subject: [PATCH 03/33] test(mi): storage-SOC LP-file equivalence; refine matrix (#744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - add test_storage_soc_lp_equivalent: the cyclic-per-period SOC "previous" built via groupby("period").roll vs an explicit positional roll produces a byte-identical LP file; a period-unaware global roll is the negative control (diverges at the period-start rows), proving the check has teeth - matrix: drop the non-operation "solve LP" row (the end-to-end compose proof is noted inline on the tracked-tests line instead); mark storage SOC 🟢 now that it is pinned by a test Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 8 ++-- test/test_mi_feasibility.py | 47 ++++++++++++++++++++ 2 files changed, 51 insertions(+), 4 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 864bb32c..c4a62c2d 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -40,8 +40,9 @@ Two axes — **feasible** (can flat+aux build an *equivalent model*?) and | ⚪ | — | parity | | 🔴 | not feasible | worse | -The 🟢 rows are tracked in `test/test_mi_feasibility.py`; PyPSA links -are pinned at **v1.2.4** (commit [`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). +The 🟢 rows are tracked in `test/test_mi_feasibility.py` (which also solves a full +LP both ways, `test_per_period_lp_equivalent`, so the per-op rewrites are shown to +compose); PyPSA links are pinned at **v1.2.4** (commit [`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). | op | MI form | flat+aux form | feasible | desirable | PyPSA call site @ v1.2.4 | |---|---|---|---|---|---| @@ -49,8 +50,7 @@ are pinned at **v1.2.4** (commit [`fb425cb`](https://github.com/PyPSA/PyPSA/tree | **level select** | `sel(snapshot=(p, slice))` | `where(period == p)` | 🟢 | ⚪ parity | [`constraints.py` L1235‑1248](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1235-L1248) (KVL; the per-period loop is topology-driven — `cycle_matrix(period)` — not MI, so flat+aux only swaps the `sns.get_loc` slice for `where`, the loop stays) | | **period roll** | `roll(1)` + `_period_start_mask` | `groupby("period").map(roll)` | 🟢 (#751) | 🟢 mask-free (boundary from grouping) | [`constraints.py` L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694) (SOC) | | **level groupby** | `groupby(MI level)` ❌ broken ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | `groupby("period").sum()` | 🟢 (#751) | 🟢 not just *nicer* — MI is *broken*, not merely workaround-y; flat+aux groups by the aux coord and just works (#751) | — | -| **solve LP** | per-period `isel`-sum `≥ d`, solved | `groupby("period").sum() ≥ d`, solved `==` | 🟢 | ⚪ parity | — | -| **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`; `where(period==…)` mask | 🔵 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | +| **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`; `where(period==…)` mask | 🟢 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | | **stochastic** | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ same mechanism as snapshot (name axis) — but PyPSA is *adding* MI here, not removing it | scenario MI [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; round-trip MI [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | | **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | | **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🔵 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | diff --git a/test/test_mi_feasibility.py b/test/test_mi_feasibility.py index 4cb42986..c0758f57 100644 --- a/test/test_mi_feasibility.py +++ b/test/test_mi_feasibility.py @@ -158,3 +158,50 @@ def test_output_restacks_to_mi() -> None: assert isinstance(restacked.indexes["snapshot"], pd.MultiIndex) assert np.array_equal(restacked.values, sol.values) assert list(restacked.to_dataframe().index.names) == ["period", "timestep"] + + +# --- storage SOC: the per-period roll, composed into a real constraint -------- # +def test_storage_soc_lp_equivalent(tmp_path) -> None: + """ + Storage SOC: flat+aux builds the byte-identical LP to an explicit per-period roll. + + Cyclic-per-period SOC -- ``soc[t] = soc[t-1] + charge[t] - demand[t]``, wrapping + within each period. ``flat`` builds the previous SOC with + ``groupby("period").roll``; ``oracle`` with an explicit per-period positional + roll (= what PyPSA's global roll + ``FILL_VALUE`` rebuild computes). A byte-equal + LP file is the whole proof -- same solver model, not just same optimum. + + The ``global`` build (period-*unaware* roll, the mistake ``FILL_VALUE`` corrects) + is the negative control: its LP differs at the two period-start rows, so the + check can fail -- it is not vacuously true. + """ + s = pd.RangeIndex(N, name="snapshot") + period = xr.DataArray(PERIOD_OF, dims="snapshot", coords={"snapshot": s}) + demand = xr.DataArray( + [1.0, 3.0, 2.0, 2.0, 1.0, 3.0], dims="snapshot", coords={"snapshot": s} + ) + prev_pos = np.empty(N, int) # per-period cyclic-previous position + for p in PERIODS: + pos = np.flatnonzero(PERIOD_OF == p) + prev_pos[pos] = np.roll(pos, 1) + + def lp(kind: str) -> str: + m = Model() + soc = m.add_variables(lower=0, coords=[s], name="soc") + charge = m.add_variables(lower=0, coords=[s], name="charge") + if kind == "flat": # per-period roll falls out of groupby + prev = (1.0 * soc).assign_coords(period=period).groupby("period") + prev = prev.roll(snapshot=1) + if "period" in prev.coords: # legacy keeps the aux coord, v1 consumes it + prev = prev.drop_vars("period") + elif kind == "oracle": # explicit per-period positional roll + prev = (1.0 * soc).isel(snapshot=prev_pos).assign_coords(snapshot=s) + else: # period-unaware global roll -- the wrong build + prev = (1.0 * soc).roll(snapshot=1) + m.add_constraints(1.0 * soc - prev - 1.0 * charge == -demand, name="soc") + path = tmp_path / f"{kind}.lp" + m.to_file(path, io_api="lp") + return path.read_text() + + assert lp("flat") == lp("oracle") # flat+aux builds the identical model + assert lp("global") != lp("oracle") # ...and the check can fail (has teeth) From b2cff46390909c3f6618214c872c26d427fa618b Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 29 Jun 2026 21:05:21 +0200 Subject: [PATCH 04/33] test(mi): cover non-cyclic SOC boundary (ramps too) (#744) Parametrize test_storage_soc_lp_equivalent over boundary in {cyclic, non-cyclic}. Non-cyclic (period start carries nothing) is the boundary ramps and non-cyclic storage use -- groupby("period").roll + a period-start where-mask vs an explicit positional roll, byte-identical LP either way. Teeth: the two boundaries yield different LPs (mask/wrap is not a no-op); the period-unaware global-roll control stays cyclic-only since non-cyclic masks away the rows it diverges on. Co-Authored-By: Claude Opus 4.8 (1M context) --- test/test_mi_feasibility.py | 41 ++++++++++++++++++++++++------------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/test/test_mi_feasibility.py b/test/test_mi_feasibility.py index c0758f57..349a7a16 100644 --- a/test/test_mi_feasibility.py +++ b/test/test_mi_feasibility.py @@ -161,31 +161,39 @@ def test_output_restacks_to_mi() -> None: # --- storage SOC: the per-period roll, composed into a real constraint -------- # -def test_storage_soc_lp_equivalent(tmp_path) -> None: +@pytest.mark.parametrize("boundary", ["cyclic", "non-cyclic"]) +def test_storage_soc_lp_equivalent(tmp_path, boundary) -> None: """ Storage SOC: flat+aux builds the byte-identical LP to an explicit per-period roll. - Cyclic-per-period SOC -- ``soc[t] = soc[t-1] + charge[t] - demand[t]``, wrapping - within each period. ``flat`` builds the previous SOC with - ``groupby("period").roll``; ``oracle`` with an explicit per-period positional - roll (= what PyPSA's global roll + ``FILL_VALUE`` rebuild computes). A byte-equal - LP file is the whole proof -- same solver model, not just same optimum. - - The ``global`` build (period-*unaware* roll, the mistake ``FILL_VALUE`` corrects) - is the negative control: its LP differs at the two period-start rows, so the - check can fail -- it is not vacuously true. + SOC dynamics ``soc[t] = soc[t-1] + charge[t] - demand[t]`` with the previous + value rolled per period -- ``cyclic`` (period start wraps to the period's last + step) or ``non-cyclic`` (no carry-over across the start, the boundary ramps and + non-cyclic storage use). ``flat`` builds the previous SOC with + ``groupby("period").roll`` (+ a period-start ``where`` mask for non-cyclic); + ``oracle`` with an explicit per-period positional roll. A byte-equal LP file is + the whole proof -- same solver model. + + Teeth: the two boundaries give *different* LPs (the mask/wrap is not a no-op); + and for ``cyclic`` a period-unaware global roll diverges at the period-start + rows. (Non-cyclic masks those rows away, so the global roll is *not* a useful + control there -- the boundary difference is.) """ s = pd.RangeIndex(N, name="snapshot") period = xr.DataArray(PERIOD_OF, dims="snapshot", coords={"snapshot": s}) demand = xr.DataArray( [1.0, 3.0, 2.0, 2.0, 1.0, 3.0], dims="snapshot", coords={"snapshot": s} ) + starts = [int(np.flatnonzero(PERIOD_OF == p)[0]) for p in PERIODS] + is_start = xr.DataArray( + np.isin(np.arange(N), starts), dims="snapshot", coords={"snapshot": s} + ) prev_pos = np.empty(N, int) # per-period cyclic-previous position for p in PERIODS: pos = np.flatnonzero(PERIOD_OF == p) prev_pos[pos] = np.roll(pos, 1) - def lp(kind: str) -> str: + def lp(kind: str, bnd: str) -> str: m = Model() soc = m.add_variables(lower=0, coords=[s], name="soc") charge = m.add_variables(lower=0, coords=[s], name="charge") @@ -198,10 +206,15 @@ def lp(kind: str) -> str: prev = (1.0 * soc).isel(snapshot=prev_pos).assign_coords(snapshot=s) else: # period-unaware global roll -- the wrong build prev = (1.0 * soc).roll(snapshot=1) + if bnd == "non-cyclic": # no carry-over across the period start + prev = prev.where(~is_start) m.add_constraints(1.0 * soc - prev - 1.0 * charge == -demand, name="soc") - path = tmp_path / f"{kind}.lp" + path = tmp_path / f"{kind}_{bnd}.lp" m.to_file(path, io_api="lp") return path.read_text() - assert lp("flat") == lp("oracle") # flat+aux builds the identical model - assert lp("global") != lp("oracle") # ...and the check can fail (has teeth) + other = "non-cyclic" if boundary == "cyclic" else "cyclic" + assert lp("flat", boundary) == lp("oracle", boundary) # flat+aux == explicit roll + assert lp("flat", boundary) != lp("flat", other) # boundary rule is real (teeth) + if boundary == "cyclic": # period-unaware roll diverges (masked away if non-cyclic) + assert lp("global", "cyclic") != lp("oracle", "cyclic") From 3033ec3a969e07aae5b7a780930a1e1b76f0bb32 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 29 Jun 2026 21:12:25 +0200 Subject: [PATCH 05/33] test(mi): separate ramp (row-drop) from non-cyclic SOC (where) (#744) The prior "ramps too" conflated two distinct period-start boundaries, now both covered as a third parametrize value: - non-cyclic storage: drop the previous *term*, keep the row (initial SOC -> rhs). This is exactly PyPSA's `previous_soc...roll(1).where(include_previous_soc)` (constraints.py L1691) -- a `where` on the term, verified byte-faithful. - ramp: drop the whole *row* at the period start via a constraint `mask=` (PyPSA's `_period_start_mask`, L838) -- no previous to ramp from. flat (groupby roll) == oracle (positional roll) byte-for-byte in all three; each boundary differs from the cyclic baseline (teeth); global-roll control stays cyclic-only since the other two mask away the rows it diverges on. Co-Authored-By: Claude Opus 4.8 (1M context) --- test/test_mi_feasibility.py | 49 ++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 20 deletions(-) diff --git a/test/test_mi_feasibility.py b/test/test_mi_feasibility.py index 349a7a16..81962d20 100644 --- a/test/test_mi_feasibility.py +++ b/test/test_mi_feasibility.py @@ -161,23 +161,28 @@ def test_output_restacks_to_mi() -> None: # --- storage SOC: the per-period roll, composed into a real constraint -------- # -@pytest.mark.parametrize("boundary", ["cyclic", "non-cyclic"]) +@pytest.mark.parametrize("boundary", ["cyclic", "non-cyclic", "ramp"]) def test_storage_soc_lp_equivalent(tmp_path, boundary) -> None: """ - Storage SOC: flat+aux builds the byte-identical LP to an explicit per-period roll. - - SOC dynamics ``soc[t] = soc[t-1] + charge[t] - demand[t]`` with the previous - value rolled per period -- ``cyclic`` (period start wraps to the period's last - step) or ``non-cyclic`` (no carry-over across the start, the boundary ramps and - non-cyclic storage use). ``flat`` builds the previous SOC with - ``groupby("period").roll`` (+ a period-start ``where`` mask for non-cyclic); - ``oracle`` with an explicit per-period positional roll. A byte-equal LP file is - the whole proof -- same solver model. - - Teeth: the two boundaries give *different* LPs (the mask/wrap is not a no-op); - and for ``cyclic`` a period-unaware global roll diverges at the period-start - rows. (Non-cyclic masks those rows away, so the global roll is *not* a useful - control there -- the boundary difference is.) + Per-period roll, composed into a constraint: flat+aux builds the byte-identical + LP to an explicit per-period roll, for every period boundary PyPSA spells. + + ``soc[t] = soc[t-1] + charge[t] - demand[t]`` with the previous value rolled per + period; only the *period start* differs (constraints.py @ v1.2.4): + + * ``cyclic`` -- wrap to the period's last step (``cyclic_state_of_charge``). + * ``non-cyclic`` -- drop the previous *term*, keep the row, initial SOC -> rhs: + PyPSA's ``previous_soc...roll(1).where(include_previous_soc)`` (L1691). A + ``where`` on the term, *not* a dropped row. + * ``ramp`` -- drop the whole *row* (no previous to ramp from): the + ``_period_start_mask`` used as a constraint ``mask`` (L838). + + ``flat`` builds the previous SOC with ``groupby("period").roll``; ``oracle`` with + an explicit per-period positional roll. A byte-equal LP file is the whole proof. + + Teeth: each boundary differs from the ``cyclic`` baseline (mask/wrap/row-drop is + not a no-op); and for ``cyclic`` a period-unaware global roll diverges at the + period-start rows (the other two mask those rows, so it is no control there). """ s = pd.RangeIndex(N, name="snapshot") period = xr.DataArray(PERIOD_OF, dims="snapshot", coords={"snapshot": s}) @@ -206,15 +211,19 @@ def lp(kind: str, bnd: str) -> str: prev = (1.0 * soc).isel(snapshot=prev_pos).assign_coords(snapshot=s) else: # period-unaware global roll -- the wrong build prev = (1.0 * soc).roll(snapshot=1) - if bnd == "non-cyclic": # no carry-over across the period start + if bnd == "non-cyclic": # drop the previous term, keep the row (PyPSA .where) prev = prev.where(~is_start) - m.add_constraints(1.0 * soc - prev - 1.0 * charge == -demand, name="soc") + con = 1.0 * soc - prev - 1.0 * charge == -demand + if bnd == "ramp": # drop the whole row at the period start (constraint mask) + m.add_constraints(con, name="soc", mask=~is_start) + else: + m.add_constraints(con, name="soc") path = tmp_path / f"{kind}_{bnd}.lp" m.to_file(path, io_api="lp") return path.read_text() - other = "non-cyclic" if boundary == "cyclic" else "cyclic" + contrast = {"cyclic": "non-cyclic", "non-cyclic": "cyclic", "ramp": "cyclic"} assert lp("flat", boundary) == lp("oracle", boundary) # flat+aux == explicit roll - assert lp("flat", boundary) != lp("flat", other) # boundary rule is real (teeth) - if boundary == "cyclic": # period-unaware roll diverges (masked away if non-cyclic) + assert lp("flat", boundary) != lp("flat", contrast[boundary]) # boundary is real + if boundary == "cyclic": # period-unaware roll diverges (masked away otherwise) assert lp("global", "cyclic") != lp("oracle", "cyclic") From 84b50e4dd25f8d9adc8785f175b8e21cf7ceec06 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 29 Jun 2026 22:49:15 +0200 Subject: [PATCH 06/33] test(mi): rename to test_period_boundary_lp_identical; show 3 boundaries (#744) The test's scope is the per-period roll's period-start boundary (cyclic / non-cyclic / ramp), not storage SOC specifically -- and it proves a byte-IDENTICAL LP file, distinct from the solve-equivalence test_per_period_lp_equivalent. Rename accordingly. Surface all three boundary spellings in the storage-SOC matrix row's flat+aux cell (wrap / .where term-drop / mask= row-drop). Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 2 +- test/test_mi_feasibility.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index c4a62c2d..dd56abae 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -50,7 +50,7 @@ compose); PyPSA links are pinned at **v1.2.4** (commit [`fb425cb`](https://githu | **level select** | `sel(snapshot=(p, slice))` | `where(period == p)` | 🟢 | ⚪ parity | [`constraints.py` L1235‑1248](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1235-L1248) (KVL; the per-period loop is topology-driven — `cycle_matrix(period)` — not MI, so flat+aux only swaps the `sns.get_loc` slice for `where`, the loop stays) | | **period roll** | `roll(1)` + `_period_start_mask` | `groupby("period").map(roll)` | 🟢 (#751) | 🟢 mask-free (boundary from grouping) | [`constraints.py` L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694) (SOC) | | **level groupby** | `groupby(MI level)` ❌ broken ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | `groupby("period").sum()` | 🟢 (#751) | 🟢 not just *nicer* — MI is *broken*, not merely workaround-y; flat+aux groups by the aux coord and just works (#751) | — | -| **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`; `where(period==…)` mask | 🟢 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | +| **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`, then period-start: wrap (cyclic) · `.where` drops the term (non-cyclic) · `mask=` drops the row (ramp) | 🟢 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | | **stochastic** | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ same mechanism as snapshot (name axis) — but PyPSA is *adding* MI here, not removing it | scenario MI [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; round-trip MI [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | | **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | | **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🔵 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | diff --git a/test/test_mi_feasibility.py b/test/test_mi_feasibility.py index 81962d20..fe91f197 100644 --- a/test/test_mi_feasibility.py +++ b/test/test_mi_feasibility.py @@ -160,9 +160,9 @@ def test_output_restacks_to_mi() -> None: assert list(restacked.to_dataframe().index.names) == ["period", "timestep"] -# --- storage SOC: the per-period roll, composed into a real constraint -------- # +# --- period-start boundary: the per-period roll composed into a constraint ---- # @pytest.mark.parametrize("boundary", ["cyclic", "non-cyclic", "ramp"]) -def test_storage_soc_lp_equivalent(tmp_path, boundary) -> None: +def test_period_boundary_lp_identical(tmp_path, boundary) -> None: """ Per-period roll, composed into a constraint: flat+aux builds the byte-identical LP to an explicit per-period roll, for every period boundary PyPSA spells. From 6659fe73e1910138b1ea3992eace654c293b3f82 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 29 Jun 2026 23:11:22 +0200 Subject: [PATCH 07/33] docs(arithmetics): stochastic is N-D, not a second MI (#744) PyPSA's stochastic path keeps `scenario` and `name` as separate xarray dims (components/array.py: expand_dims(scenario=...), isel(scenario=0)); the `(scenario, name)` MultiIndex is manufactured only by `.stack(combined=...)` at the pandas output boundary, and common.py's MI is the input DataFrame's columns. So it is not an in-model MI like the snapshot `(period, timestep)` index -- it is already the flat+aux N-D form, nothing to decompose. Correct the stochastic row's verdict and the conclusion/decision-record prose: the one real open item is the public `n.snapshots` choice; stochastic drops from "genuine open risk / second MI" to a low-risk watch on whether #1484 ever introduces an in-model stacked index (v1.2.4 does not). Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 38 +++++++++++--------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index dd56abae..27338f19 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -51,7 +51,7 @@ compose); PyPSA links are pinned at **v1.2.4** (commit [`fb425cb`](https://githu | **period roll** | `roll(1)` + `_period_start_mask` | `groupby("period").map(roll)` | 🟢 (#751) | 🟢 mask-free (boundary from grouping) | [`constraints.py` L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694) (SOC) | | **level groupby** | `groupby(MI level)` ❌ broken ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | `groupby("period").sum()` | 🟢 (#751) | 🟢 not just *nicer* — MI is *broken*, not merely workaround-y; flat+aux groups by the aux coord and just works (#751) | — | | **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`, then period-start: wrap (cyclic) · `.where` drops the term (non-cyclic) · `mask=` drops the row (ramp) | 🟢 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | -| **stochastic** | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ same mechanism as snapshot (name axis) — but PyPSA is *adding* MI here, not removing it | scenario MI [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; round-trip MI [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | +| **stochastic** | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ *not* an in-model MI — `scenario`/`name` are separate dims (N-D); the `(scenario, name)` MI is only `.stack`→pandas at output. Nothing to decompose; already the flat+aux target form, PyPSA's MI here is output-cosmetic | `(scenario,name)` *pandas cols* [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; output `.stack(scenario,name)` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | | **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | | **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🔵 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | | **n.snapshots** | `pd.MultiIndex` public API | flat dim + level coords | 🔵 | 🔴 PyPSA API migration | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | @@ -89,13 +89,15 @@ conclusion for linopy is clean: **accept MI as input sugar, decompose on entry (`reset_index`), and never reconstruct — flat in, flat out. linopy drops MultiIndex from its mental model entirely.** -The remaining cost is **not inside linopy**: it is (a) whether PyPSA keeps -`n.snapshots` as MI — a cheap boundary wrap it can do on its own side, decoupled -from this decision — and (b) the **stochastic / Monte-Carlo direction** -(`stochastic` row), a *second* MI `(scenario, name)` that PyPSA is actively -entrenching ([#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants an MI level -per sampled dimension). That is the genuine open risk, and it is a PyPSA-side -call — linopy works either way. +The remaining cost is **not inside linopy**, and it is essentially **one** thing: +whether PyPSA keeps `n.snapshots` as MI — a cheap boundary wrap it can do on its +own side, decoupled from this decision. The **stochastic / Monte-Carlo direction** +(`stochastic` row) turns out *not* to be a second in-model MI at all: `scenario` +and `name` are separate N-D dims inside linopy, and the `(scenario, name)` MI is +only an `xarray.stack` at the pandas output. So there is nothing to decompose — +it is already the flat+aux form. The one thing to watch is whether future +Monte-Carlo work ([#1484](https://github.com/PyPSA/PyPSA/issues/1484)) introduces an +*in-model* stacked index rather than more dims; nothing in v1.2.4 does. **Side finding (not a row):** `Variable.sel` can't MI-tuple-select (`x.sel(snapshot=(p, slice))` → `InvalidIndexError`), which is why PyPSA drops to @@ -146,15 +148,16 @@ the growth `reindex_like(lhs.data)` linked above exist because MI groupby is bro pydata/xarray#6836). #751 fixed the level groupby, so flat+aux **deletes** those specific reaches rather than re-spelling them. These build-time sites `.sel` the *stored* MI mid-build, so they migrate whether or not PyPSA re-stacks the output. -What's left to decide is only the **public `n.snapshots`** question and the -stochastic 2nd MI (`n.snapshots`/`stochastic` rows), pressure-tested against -[PyPSA#1484](https://github.com/PyPSA/PyPSA/issues/1484) (Monte-Carlo, which wants -an MI level per sampled dimension). +What's left to decide is only the **public `n.snapshots`** question (`n.snapshots` +row). Stochastic is *not* a second in-model MI — `scenario`/`name` are separate +N-D dims, the MI is pandas-output only — so the only watch item is whether +[PyPSA#1484](https://github.com/PyPSA/PyPSA/issues/1484) (Monte-Carlo) ever turns +that into an in-model stacked index; v1.2.4 does not. ## Decision record (to fill once the open rows close) -> _The `n.snapshots` scope, the stochastic-MI resolution, and the evidence rows -> that justify them. This note, once complete, is what closes #744._ +> _The `n.snapshots` scope (the one real open item) and the evidence rows that +> justify it. This note, once complete, is what closes #744._ - **linopy drops MultiIndex from its mental model** — accept MI as input sugar, decompose on entry (`reset_index`), never reconstruct (flat in, flat out). @@ -164,6 +167,7 @@ an MI level per sampled dimension). PyPSA owns (`output` row, tested); no MI adapter lives inside linopy. - **`n.snapshots`** — PyPSA's independent, decoupled choice: keep MI (wrap at its own boundary) or flatten. _TBD, PyPSA-side._ -- **Stochastic 2nd MI `(scenario, name)`** (`stochastic` row) — the genuine open risk; - PyPSA is entrenching it ([#1484](https://github.com/PyPSA/PyPSA/issues/1484)). - Resolve via a PyPSA-side solution-equivalence spike. _TBD._ +- **Stochastic is N-D, not a second MI** (`stochastic` row) — `scenario`/`name` are + separate dims; the `(scenario, name)` MI is only an output `.stack`. No in-model MI + to remove. Watch only whether [#1484](https://github.com/PyPSA/PyPSA/issues/1484) + (Monte-Carlo) introduces an in-model stacked index; v1.2.4 does not. _Low risk._ From 34420d7039ac4a518822544c12f50171393ecb31 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 29 Jun 2026 23:16:01 +0200 Subject: [PATCH 08/33] =?UTF-8?q?docs(arithmetics):=20tag=20boundary=20/?= =?UTF-8?q?=20non-linopy-MI=20rows=20with=20=E2=8A=A5=20(#744)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six rows are the genuine in-linopy MultiIndex decision (snapshot (period, timestep) sitting inside linopy variables). The other three are not, and now carry a ⊥ tag + legend: output (PyPSA-side boundary re-stack), n.snapshots (PyPSA public API, never a linopy model index), stochastic (not an MI at all — N-D dims). Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 27338f19..013d6028 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -44,6 +44,11 @@ The 🟢 rows are tracked in `test/test_mi_feasibility.py` (which also solves a LP both ways, `test_per_period_lp_equivalent`, so the per-op rewrites are shown to compose); PyPSA links are pinned at **v1.2.4** (commit [`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). +**⊥** marks a row that is *not* an in-linopy MultiIndex: the MI lives only at the +PyPSA/pandas boundary, or — for `stochastic` — nowhere in the model. The six +unmarked rows are the linopy decision proper (the snapshot `(period, timestep)` MI +that actually sits inside linopy variables); the ⊥ rows are PyPSA-side or no-op. + | op | MI form | flat+aux form | feasible | desirable | PyPSA call site @ v1.2.4 | |---|---|---|---|---|---| | **entry** | `coords=[mi]` | `reset_index(dim)` | 🟢 | 🟢 deletes MI machinery | [`constraints.py` L1052](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1052) (`from_pandas_multiindex`) | @@ -51,10 +56,10 @@ compose); PyPSA links are pinned at **v1.2.4** (commit [`fb425cb`](https://githu | **period roll** | `roll(1)` + `_period_start_mask` | `groupby("period").map(roll)` | 🟢 (#751) | 🟢 mask-free (boundary from grouping) | [`constraints.py` L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694) (SOC) | | **level groupby** | `groupby(MI level)` ❌ broken ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | `groupby("period").sum()` | 🟢 (#751) | 🟢 not just *nicer* — MI is *broken*, not merely workaround-y; flat+aux groups by the aux coord and just works (#751) | — | | **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`, then period-start: wrap (cyclic) · `.where` drops the term (non-cyclic) · `mask=` drops the row (ramp) | 🟢 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | -| **stochastic** | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ *not* an in-model MI — `scenario`/`name` are separate dims (N-D); the `(scenario, name)` MI is only `.stack`→pandas at output. Nothing to decompose; already the flat+aux target form, PyPSA's MI here is output-cosmetic | `(scenario,name)` *pandas cols* [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; output `.stack(scenario,name)` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | -| **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | +| **stochastic** ⊥ *not an MI* | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ *not* an in-model MI — `scenario`/`name` are separate dims (N-D); the `(scenario, name)` MI is only `.stack`→pandas at output. Nothing to decompose; already the flat+aux target form, PyPSA's MI here is output-cosmetic | `(scenario,name)` *pandas cols* [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; output `.stack(scenario,name)` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | +| **output** ⊥ *boundary* | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | | **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🔵 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | -| **n.snapshots** | `pd.MultiIndex` public API | flat dim + level coords | 🔵 | 🔴 PyPSA API migration | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | +| **n.snapshots** ⊥ *PyPSA API* | `pd.MultiIndex` public API | flat dim + level coords | 🔵 | 🔴 PyPSA API migration | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | **No row needs a linopy change to be feasible.** The `entry` conversion is a user-side `reset_index` with today's linopy; linopy auto-accepting `coords=[mi]` From 5f4e0aa9abdf8465a5caff83653d9c7ada652a4b Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 29 Jun 2026 23:21:24 +0200 Subject: [PATCH 09/33] docs(arithmetics): lead with the MI-scope premise; add reset_index root cause (#744) Open with a bold, falsifiable claim: inside the linopy model PyPSA uses a MultiIndex in exactly one place -- the snapshot dim (period, timestep). The lookalikes are ruled out inline (stochastic = N-D dims, n.snapshots = public API, output = pandas .stack), so the whole linopy-side question collapses to one axis; other axes are assumed N-D, not audited -- corrections welcome. Add a "What reset_index changes (and doesn't)" section stating the root cause: snapshot is one dim whose index is a MultiIndex, period/timestep are levels (never dims, surfacing only via .sel-collapse), and reset_index flips only the index type -- not the dim count -- turning collapse/loop/.data into direct groupby/where. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 33 ++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 013d6028..fbef7e0b 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -7,6 +7,27 @@ > in [`test/test_mi_feasibility.py`](../test/test_mi_feasibility.py) (runs under > both `legacy` and `v1`). +## Where MultiIndex is used in PyPSA + +**Working assumption, stated boldly so it can be falsified: inside the linopy model +PyPSA uses a `pd.MultiIndex` in _exactly one place_ — the `snapshot` dimension** +(multi-period `(period, timestep)`). Everything else that looks MultiIndex-ish is +*not* an in-model index — which is why only those rows below carry a ⊥ tag: + +- **`stochastic` `(scenario, name)`** — `scenario` and `name` are separate N-D dims + inside linopy; the MI is only an `xarray.stack` at the pandas output + (`components/array.py` @ v1.2.4). Not an in-model MI. +- **`n.snapshots`** — a real MI, but it lives in PyPSA's *public API*, never handed + to linopy as a model index. +- **`output` / `dual`** — MI appears only on the returned pandas object, rebuilt at + the boundary by the caller. + +So the whole linopy-side question collapses to **one axis, `snapshot`**, and the +rest of this note works from that premise. Other axes (contingencies, a future +Monte-Carlo #1484) are *assumed* to follow the same N-D pattern, not audited — if +PyPSA holds an in-model MultiIndex anywhere else, this assumption and the scope +below need revisiting. **Corrections welcome.** + **Question.** Can v1 drop the stacked `pd.MultiIndex` snapshot for a flat `snapshot` dim carrying `period`/`timestep` as auxiliary level coords? @@ -115,6 +136,18 @@ rows are **PyPSA-owned** and answer the **transition** question — verified by solution-equivalence on real networks (multi-period, stochastic, Monte-Carlo) plus scoping the public `n.snapshots` change. +## What `reset_index` changes (and doesn't) + +Today `snapshot` is a **single dimension** whose *index* is a `MultiIndex(period, +timestep)`; `period`/`timestep` are **levels, never dimensions**. They surface as a +real dim only *momentarily* — `.sel(period=p)` collapses the 2-level MI to one level +and xarray renames `snapshot → timestep`; the two never coexist as independent dims. +That is exactly why per-period code must `.sel`-collapse, loop, or reach into `.data` +(the SOC/KVL/objective coupling). `reset_index("snapshot")` flips **only** the index +type (MultiIndex → flat), not the dim count — `('snapshot', 'name')` before and +after — demoting the levels to ordinary aux coords; collapse/loop/`.data` then becomes +direct `groupby`/`where`. That one line *is* the whole flat+aux transformation. + ## Sub-decision: the snapshot dim coordinate After `reset_index`, the dim is **coordinate-less** (xarray virtualizes `0..N-1` From 6ca96db6c481110bb0b3f0011df5c41a19499a99 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 29 Jun 2026 23:22:03 +0200 Subject: [PATCH 10/33] docs(arithmetics): state the snapshot-only MI scope plainly (#744) Drop the "working assumption / corrections welcome" framing -- just state that inside the linopy model MultiIndex is used only on the snapshot dim, with the three lookalikes ruled out inline. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index fbef7e0b..de49fc9b 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -9,24 +9,20 @@ ## Where MultiIndex is used in PyPSA -**Working assumption, stated boldly so it can be falsified: inside the linopy model -PyPSA uses a `pd.MultiIndex` in _exactly one place_ — the `snapshot` dimension** -(multi-period `(period, timestep)`). Everything else that looks MultiIndex-ish is -*not* an in-model index — which is why only those rows below carry a ⊥ tag: +Inside the linopy model, PyPSA uses a `pd.MultiIndex` in **exactly one place — the +`snapshot` dimension** (multi-period `(period, timestep)`). Everything else that +looks MultiIndex-ish is *not* an in-model index — which is why only those rows below +carry a ⊥ tag: - **`stochastic` `(scenario, name)`** — `scenario` and `name` are separate N-D dims inside linopy; the MI is only an `xarray.stack` at the pandas output - (`components/array.py` @ v1.2.4). Not an in-model MI. -- **`n.snapshots`** — a real MI, but it lives in PyPSA's *public API*, never handed - to linopy as a model index. + (`components/array.py` @ v1.2.4). +- **`n.snapshots`** — a real MI, but in PyPSA's *public API*, never handed to linopy + as a model index. - **`output` / `dual`** — MI appears only on the returned pandas object, rebuilt at the boundary by the caller. -So the whole linopy-side question collapses to **one axis, `snapshot`**, and the -rest of this note works from that premise. Other axes (contingencies, a future -Monte-Carlo #1484) are *assumed* to follow the same N-D pattern, not audited — if -PyPSA holds an in-model MultiIndex anywhere else, this assumption and the scope -below need revisiting. **Corrections welcome.** +So the whole linopy-side question is **one axis, `snapshot`**. **Question.** Can v1 drop the stacked `pd.MultiIndex` snapshot for a flat `snapshot` dim carrying `period`/`timestep` as auxiliary level coords? From d7ce3fc63f8ea8691bd1553bda2ddbdbdde64ce3 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 18:47:59 +0200 Subject: [PATCH 11/33] =?UTF-8?q?docs(arithmetics):=20output=20is=20in-lin?= =?UTF-8?q?opy=20MI=20(snapshot=20out),=20not=20=E2=8A=A5=20boundary=20(#7?= =?UTF-8?q?44)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit solution/dual carry the snapshot dim, which today is the MultiIndex inherited from the MI-indexed variables -- so output holds the same in-model snapshot MI as entry, on the way out. Drop the ⊥ tag; show the snapshot MI flowing in (entry) -> ops -> out (output) in the lead and legend. Only stochastic (N-D) and n.snapshots (public API) remain ⊥. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index de49fc9b..96e67c1a 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -10,17 +10,17 @@ ## Where MultiIndex is used in PyPSA Inside the linopy model, PyPSA uses a `pd.MultiIndex` in **exactly one place — the -`snapshot` dimension** (multi-period `(period, timestep)`). Everything else that -looks MultiIndex-ish is *not* an in-model index — which is why only those rows below -carry a ⊥ tag: +`snapshot` dimension** (multi-period `(period, timestep)`). It rides that one axis +through the whole lifecycle: in at `entry`, through the per-period ops, *out* at +`output` (the `solution`/`dual` carry the `snapshot` dim, so they are MI-indexed +too), and parked on `snapshots param`. Two things that merely *look* MultiIndex-ish +are not an in-model index — they carry the ⊥ tag below: - **`stochastic` `(scenario, name)`** — `scenario` and `name` are separate N-D dims inside linopy; the MI is only an `xarray.stack` at the pandas output (`components/array.py` @ v1.2.4). - **`n.snapshots`** — a real MI, but in PyPSA's *public API*, never handed to linopy as a model index. -- **`output` / `dual`** — MI appears only on the returned pandas object, rebuilt at - the boundary by the caller. So the whole linopy-side question is **one axis, `snapshot`**. @@ -61,10 +61,10 @@ The 🟢 rows are tracked in `test/test_mi_feasibility.py` (which also solves a LP both ways, `test_per_period_lp_equivalent`, so the per-op rewrites are shown to compose); PyPSA links are pinned at **v1.2.4** (commit [`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). -**⊥** marks a row that is *not* an in-linopy MultiIndex: the MI lives only at the -PyPSA/pandas boundary, or — for `stochastic` — nowhere in the model. The six -unmarked rows are the linopy decision proper (the snapshot `(period, timestep)` MI -that actually sits inside linopy variables); the ⊥ rows are PyPSA-side or no-op. +**⊥** marks the two rows that are *not* an in-linopy MultiIndex: `stochastic` (N-D +dims, no MI in the model) and `n.snapshots` (PyPSA's public-API index, never handed +to linopy). The other seven rows are the one `snapshot` `(period, timestep)` MI as +it lives inside linopy — in at `entry`, through the ops, out at `output`. | op | MI form | flat+aux form | feasible | desirable | PyPSA call site @ v1.2.4 | |---|---|---|---|---|---| @@ -74,7 +74,7 @@ that actually sits inside linopy variables); the ⊥ rows are PyPSA-side or no-o | **level groupby** | `groupby(MI level)` ❌ broken ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | `groupby("period").sum()` | 🟢 (#751) | 🟢 not just *nicer* — MI is *broken*, not merely workaround-y; flat+aux groups by the aux coord and just works (#751) | — | | **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`, then period-start: wrap (cyclic) · `.where` drops the term (non-cyclic) · `mask=` drops the row (ramp) | 🟢 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | | **stochastic** ⊥ *not an MI* | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ *not* an in-model MI — `scenario`/`name` are separate dims (N-D); the `(scenario, name)` MI is only `.stack`→pandas at output. Nothing to decompose; already the flat+aux target form, PyPSA's MI here is output-cosmetic | `(scenario,name)` *pandas cols* [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; output `.stack(scenario,name)` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | -| **output** ⊥ *boundary* | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | +| **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | | **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🔵 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | | **n.snapshots** ⊥ *PyPSA API* | `pd.MultiIndex` public API | flat dim + level coords | 🔵 | 🔴 PyPSA API migration | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | From aa5df1710278a747bd1b1927798b6ae2fd5f0baf Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 18:50:38 +0200 Subject: [PATCH 12/33] =?UTF-8?q?test(mi):=20pin=20snapshots-param=20flat?= =?UTF-8?q?=20rebuild;=20matrix=20=F0=9F=94=B5=E2=86=92=F0=9F=9F=A2=20(#74?= =?UTF-8?q?4)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PyPSA parks the snapshot index on model.parameters (assign(snapshots=sns), optimize.py L689) and reads it back via .snapshots.to_index() (L905) -- today that lands a real MultiIndex *inside* the linopy object. test_snapshots_param_flat_rebuild shows the flat+aux form parks only a flat snapshot + period/timestep aux vars (no MI index inside parameters) and rebuilds the identical index on demand. Flips the last in-linopy 🔵 row to 🟢. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 2 +- test/test_mi_feasibility.py | 34 ++++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 96e67c1a..1cc81e59 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -75,7 +75,7 @@ it lives inside linopy — in at `entry`, through the ops, out at `output`. | **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`, then period-start: wrap (cyclic) · `.where` drops the term (non-cyclic) · `mask=` drops the row (ramp) | 🟢 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | | **stochastic** ⊥ *not an MI* | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ *not* an in-model MI — `scenario`/`name` are separate dims (N-D); the `(scenario, name)` MI is only `.stack`→pandas at output. Nothing to decompose; already the flat+aux target form, PyPSA's MI here is output-cosmetic | `(scenario,name)` *pandas cols* [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; output `.stack(scenario,name)` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | | **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | -| **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🔵 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | +| **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🟢 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | | **n.snapshots** ⊥ *PyPSA API* | `pd.MultiIndex` public API | flat dim + level coords | 🔵 | 🔴 PyPSA API migration | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | **No row needs a linopy change to be feasible.** The `entry` conversion is a diff --git a/test/test_mi_feasibility.py b/test/test_mi_feasibility.py index fe91f197..10b2bf75 100644 --- a/test/test_mi_feasibility.py +++ b/test/test_mi_feasibility.py @@ -227,3 +227,37 @@ def lp(kind: str, bnd: str) -> str: assert lp("flat", boundary) != lp("flat", contrast[boundary]) # boundary is real if boundary == "cyclic": # period-unaware roll diverges (masked away otherwise) assert lp("global", "cyclic") != lp("oracle", "cyclic") + + +# --- snapshots param: the MI PyPSA parks inside a linopy object --------------- # +def test_snapshots_param_flat_rebuild() -> None: + """ + Snapshots param: the snapshot index PyPSA stores on ``model.parameters`` need + not be an MI -- a flat `snapshot` + `period`/`timestep` aux vars rebuild it. + + PyPSA parks `model.parameters.assign(snapshots=sns)` and reads it back with + `parameters.snapshots.to_index()` (optimize.py L689 / L905). Today that lands a + real `MultiIndex` *inside* the linopy object; flat+aux parks only flat + `snapshot` + level vars and rebuilds the identical index on demand. + """ + mi = _mi() + + # MI way (PyPSA today): the MI lives inside model.parameters, read back verbatim + m = Model() + m.parameters = m.parameters.assign(snapshots=mi) # optimize.py L689 + assert isinstance(m.parameters.indexes["snapshots"], pd.MultiIndex) + assert m.parameters.snapshots.to_index().equals(mi) # optimize.py L905 + + # flat+aux way: park flat snapshot + level vars; no MI inside the object + m2 = Model() + flat = _flat() + m2.parameters = m2.parameters.assign( + period=flat["period"], timestep=flat["timestep"] + ) + assert isinstance(m2.parameters.indexes["snapshot"], pd.RangeIndex) + assert "snapshots" not in m2.parameters # no MI parked inside the linopy object + rebuilt = pd.MultiIndex.from_arrays( + [m2.parameters.period.values, m2.parameters.timestep.values], + names=["period", "timestep"], + ) + assert rebuilt.equals(mi) # same index, rebuilt from the aux vars From 3b8e085d291cabb868f100eb8b4167e9f2c1b9aa Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 18:52:22 +0200 Subject: [PATCH 13/33] =?UTF-8?q?docs(arithmetics):=20sink=20the=20?= =?UTF-8?q?=E2=8A=A5=20boundary=20rows=20to=20the=20bottom=20(#744)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Order the matrix as the seven in-linopy snapshot-MI rows first (entry → ops → storage SOC → output → snapshots param), then the two ⊥ rows that are outside linopy (stochastic, n.snapshots) last. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 1cc81e59..974bf3b0 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -73,9 +73,9 @@ it lives inside linopy — in at `entry`, through the ops, out at `output`. | **period roll** | `roll(1)` + `_period_start_mask` | `groupby("period").map(roll)` | 🟢 (#751) | 🟢 mask-free (boundary from grouping) | [`constraints.py` L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694) (SOC) | | **level groupby** | `groupby(MI level)` ❌ broken ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | `groupby("period").sum()` | 🟢 (#751) | 🟢 not just *nicer* — MI is *broken*, not merely workaround-y; flat+aux groups by the aux coord and just works (#751) | — | | **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`, then period-start: wrap (cyclic) · `.where` drops the term (non-cyclic) · `mask=` drops the row (ramp) | 🟢 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | -| **stochastic** ⊥ *not an MI* | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ *not* an in-model MI — `scenario`/`name` are separate dims (N-D); the `(scenario, name)` MI is only `.stack`→pandas at output. Nothing to decompose; already the flat+aux target form, PyPSA's MI here is output-cosmetic | `(scenario,name)` *pandas cols* [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; output `.stack(scenario,name)` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | | **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | | **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🟢 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | +| **stochastic** ⊥ *not an MI* | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ *not* an in-model MI — `scenario`/`name` are separate dims (N-D); the `(scenario, name)` MI is only `.stack`→pandas at output. Nothing to decompose; already the flat+aux target form, PyPSA's MI here is output-cosmetic | `(scenario,name)` *pandas cols* [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; output `.stack(scenario,name)` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | | **n.snapshots** ⊥ *PyPSA API* | `pd.MultiIndex` public API | flat dim + level coords | 🔵 | 🔴 PyPSA API migration | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | **No row needs a linopy change to be feasible.** The `entry` conversion is a From 7ef42d8f1342adc59a7c67b4d1d0b9dfc4a2f81e Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 18:58:53 +0200 Subject: [PATCH 14/33] docs(arithmetics): split PyPSA-side MI usages into their own table (#744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The feasibility matrix is now exactly the seven in-linopy snapshot-MI rows (all 🟢). The two ⊥ rows move to a dedicated section, "Observed PyPSA MultiIndex usages — not linopy's to solve": stochastic and n.snapshots are observed PyPSA usages that provably need no in-linopy solution and are cheap for PyPSA to handle at its boundary. They fail the in-linopy-MI test on opposite axes -- stochastic is inside linopy but not an MI; n.snapshots is an MI but not inside linopy -- which the new table states directly. Drop the now-unused ⊥ tag/legend. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 41 +++++++++++--------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 974bf3b0..520b1aaa 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -14,7 +14,8 @@ Inside the linopy model, PyPSA uses a `pd.MultiIndex` in **exactly one place — through the whole lifecycle: in at `entry`, through the per-period ops, *out* at `output` (the `solution`/`dual` carry the `snapshot` dim, so they are MI-indexed too), and parked on `snapshots param`. Two things that merely *look* MultiIndex-ish -are not an in-model index — they carry the ⊥ tag below: +are not an in-model index — they are observed PyPSA usages, handled PyPSA-side, in +their own table after the matrix: - **`stochastic` `(scenario, name)`** — `scenario` and `name` are separate N-D dims inside linopy; the MI is only an `xarray.stack` at the pandas output @@ -61,10 +62,9 @@ The 🟢 rows are tracked in `test/test_mi_feasibility.py` (which also solves a LP both ways, `test_per_period_lp_equivalent`, so the per-op rewrites are shown to compose); PyPSA links are pinned at **v1.2.4** (commit [`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). -**⊥** marks the two rows that are *not* an in-linopy MultiIndex: `stochastic` (N-D -dims, no MI in the model) and `n.snapshots` (PyPSA's public-API index, never handed -to linopy). The other seven rows are the one `snapshot` `(period, timestep)` MI as -it lives inside linopy — in at `entry`, through the ops, out at `output`. +All seven rows are the one `snapshot` `(period, timestep)` MI as it lives inside +linopy — in at `entry`, through the ops, out at `output`, parked on `snapshots +param` — and all are 🟢 (tested under both semantics). | op | MI form | flat+aux form | feasible | desirable | PyPSA call site @ v1.2.4 | |---|---|---|---|---|---| @@ -75,8 +75,6 @@ it lives inside linopy — in at `entry`, through the ops, out at `output`. | **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`, then period-start: wrap (cyclic) · `.where` drops the term (non-cyclic) · `mask=` drops the row (ramp) | 🟢 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | | **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | | **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🟢 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | -| **stochastic** ⊥ *not an MI* | `scenario` is a clean dim *into* linopy; `(scenario, name)` MI only on the pandas round-trip | `scenario` dim unchanged; round-trip rebuild like `output` | 🔵 | ⚪ *not* an in-model MI — `scenario`/`name` are separate dims (N-D); the `(scenario, name)` MI is only `.stack`→pandas at output. Nothing to decompose; already the flat+aux target form, PyPSA's MI here is output-cosmetic | `(scenario,name)` *pandas cols* [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); feature [PyPSA#1154](https://github.com/PyPSA/PyPSA/pull/1154), [#1484](https://github.com/PyPSA/PyPSA/issues/1484) wants *more*; output `.stack(scenario,name)` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64) | -| **n.snapshots** ⊥ *PyPSA API* | `pd.MultiIndex` public API | flat dim + level coords | 🔵 | 🔴 PyPSA API migration | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | **No row needs a linopy change to be feasible.** The `entry` conversion is a user-side `reset_index` with today's linopy; linopy auto-accepting `coords=[mi]` @@ -111,15 +109,20 @@ conclusion for linopy is clean: **accept MI as input sugar, decompose on entry (`reset_index`), and never reconstruct — flat in, flat out. linopy drops MultiIndex from its mental model entirely.** -The remaining cost is **not inside linopy**, and it is essentially **one** thing: -whether PyPSA keeps `n.snapshots` as MI — a cheap boundary wrap it can do on its -own side, decoupled from this decision. The **stochastic / Monte-Carlo direction** -(`stochastic` row) turns out *not* to be a second in-model MI at all: `scenario` -and `name` are separate N-D dims inside linopy, and the `(scenario, name)` MI is -only an `xarray.stack` at the pandas output. So there is nothing to decompose — -it is already the flat+aux form. The one thing to watch is whether future -Monte-Carlo work ([#1484](https://github.com/PyPSA/PyPSA/issues/1484)) introduces an -*in-model* stacked index rather than more dims; nothing in v1.2.4 does. +## Observed PyPSA MultiIndex usages — not linopy's to solve + +Two further MultiIndex usages show up in PyPSA but **do not need solving inside +linopy** — they're cheap for PyPSA to handle at its own boundary. Each fails the +*in-linopy MI* test, on **opposite** axes: `stochastic` is inside linopy but isn't +an MI; `n.snapshots` is an MI but isn't inside linopy. + +| PyPSA MI usage | inside linopy? | an MI? | why linopy needn't solve it | call site @ v1.2.4 | +|---|---|---|---|---| +| **stochastic** `(scenario, name)` | **yes** — `scenario` is a real dim (N-D) | **no** — only an `xarray.stack` at the pandas output | already the flat+aux shape in the model; the MI is output-cosmetic, rebuilt at the boundary like `output`. Watch only whether [#1484](https://github.com/PyPSA/PyPSA/issues/1484) ever makes it an *in-model* stacked index (v1.2.4 does not) | `(scenario,name)` pandas cols [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); output `.stack` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64); [#1154](https://github.com/PyPSA/PyPSA/pull/1154) | +| **n.snapshots** | **no** — a PyPSA `Network` attribute | **yes** — but the MI lives in PyPSA | linopy only `reindex_like`s against it; the MI never enters the model. Keep MI (wrap at its boundary) or flatten — PyPSA's call, decoupled from this decision | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | + +So the remaining cost is **not inside linopy**, and it is essentially **one** thing: +whether PyPSA keeps `n.snapshots` as MI — a cheap boundary wrap on its own side. **Side finding (not a row):** `Variable.sel` can't MI-tuple-select (`x.sel(snapshot=(p, slice))` → `InvalidIndexError`), which is why PyPSA drops to @@ -127,9 +130,9 @@ Monte-Carlo work ([#1484](https://github.com/PyPSA/PyPSA/issues/1484)) introduce becomes `where(period == p)` / `isel`, removing the internals reach. Pinned by `test_variable_mi_tuple_sel_not_forwarded`. -The 🟢 rows answer the **steady-state (linopy)** question; the 🔵 -rows are **PyPSA-owned** and answer the **transition** question — verified by -solution-equivalence on real networks (multi-period, stochastic, Monte-Carlo) +The matrix (all 🟢) answers the **steady-state (linopy)** question — tested under +both semantics. The **transition** is PyPSA-side: the two usages in the table +above, verified by solution-equivalence on real networks (multi-period, stochastic) plus scoping the public `n.snapshots` change. ## What `reset_index` changes (and doesn't) From de1f6eb5654b1dd395a8ad3543ded5b05f4e013a Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 19:03:20 +0200 Subject: [PATCH 15/33] docs(arithmetics): BLUF-first rewrite; fix glyph overload; condense (#744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three structural fixes from review, one coherent pass: - Lead with a Verdict (BLUF): feasible with zero linopy changes, flat in/flat out, one open PyPSA-side item (n.snapshots). Everything after is evidence for a claim the reader already holds, instead of arguing up to it. - Fix the per-column glyph overload: feasibility uses glyphs (✅ tested / 🔲 achievable / ❌ no), desirable uses plain words (better / parity / worse) -- a glyph no longer means two things across columns. - Hoist the settled-vs-open framing above the matrix (Scope + the two-table split already embody it), drop the now-redundant decoder paragraph and restated conclusion. Net 211 → 160 lines; same evidence and call sites, less argument. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 258 ++++++++----------- 1 file changed, 104 insertions(+), 154 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 520b1aaa..f135a6ae 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -1,36 +1,34 @@ # MultiIndex feasibility for v1 (#744) -> Verification note (Claude Code, prompted by @FBumann), 2026-06-29. Tracks -> whether linopy can drop **first-class `pd.MultiIndex`** support in v1 in favour -> of a **flat dim + auxiliary level coords** model. Discussion home: -> [#744](https://github.com/PyPSA/linopy/issues/744). Equality checks are tracked -> in [`test/test_mi_feasibility.py`](../test/test_mi_feasibility.py) (runs under -> both `legacy` and `v1`). - -## Where MultiIndex is used in PyPSA - -Inside the linopy model, PyPSA uses a `pd.MultiIndex` in **exactly one place — the -`snapshot` dimension** (multi-period `(period, timestep)`). It rides that one axis -through the whole lifecycle: in at `entry`, through the per-period ops, *out* at -`output` (the `solution`/`dual` carry the `snapshot` dim, so they are MI-indexed -too), and parked on `snapshots param`. Two things that merely *look* MultiIndex-ish -are not an in-model index — they are observed PyPSA usages, handled PyPSA-side, in -their own table after the matrix: - -- **`stochastic` `(scenario, name)`** — `scenario` and `name` are separate N-D dims - inside linopy; the MI is only an `xarray.stack` at the pandas output - (`components/array.py` @ v1.2.4). -- **`n.snapshots`** — a real MI, but in PyPSA's *public API*, never handed to linopy - as a model index. - -So the whole linopy-side question is **one axis, `snapshot`**. - -**Question.** Can v1 drop the stacked `pd.MultiIndex` snapshot for a flat -`snapshot` dim carrying `period`/`timestep` as auxiliary level coords? - -**"Feasible"** has a precise, testable meaning: every real MI use case has a -flat+aux form that builds an *equivalent model* — proven by an explicit equality -check, not asserted. +> Verification note (Claude Code, prompted by @FBumann), 2026-06-29. Tracks whether +> linopy can drop **first-class `pd.MultiIndex`** support in v1 for a **flat dim + +> auxiliary level coords** model. Home: [#744](https://github.com/PyPSA/linopy/issues/744). +> Equality checks: [`test/test_mi_feasibility.py`](../test/test_mi_feasibility.py) +> (runs under both `legacy` and `v1`). + +## Verdict + +**Feasible — yes, with zero linopy changes.** linopy accepts an MI snapshot as +*input sugar*, `reset_index` on entry, and goes **flat in / flat out** — never +reconstructing it. PyPSA uses a MultiIndex on exactly one axis, `snapshot` +`(period, timestep)`, and all seven of its in-linopy uses have a flat+aux form that +builds the identical model, each tested under both semantics. + +**One open item, and it is PyPSA's, not linopy's:** whether `n.snapshots` stays a +MultiIndex — a cheap boundary wrap PyPSA owns. + +Everything below is evidence: the **matrix** of the seven in-linopy ops (settled, +tested now), then the two **PyPSA-side usages** (the transition, PyPSA-owned). + +## Scope: one axis, `snapshot` + +Inside the linopy model PyPSA uses a `pd.MultiIndex` only on the `snapshot` +dimension; it rides that axis through the whole lifecycle — in at `entry`, through +the per-period ops, out at `output` (`solution`/`dual` carry `snapshot`, so they are +MI-indexed too), parked on `snapshots param`. Two usages merely *look* MultiIndex-ish +and are handled PyPSA-side (their own table below): `stochastic` (`scenario`/`name` +are separate N-D dims; the MI is only an `xarray.stack` at the pandas output) and +`n.snapshots` (a real MI, but PyPSA's *public API*, never a linopy model index). ## Data model under test @@ -42,99 +40,71 @@ check, not asserted. | alignment | tuple-identity | **positional** (one canonical snapshot order) | The entry conversion is byte-identical across linopy's supported xarray range -(2024.2.0 floor → 2026.4.0); it post-dates xarray's explicit-indexes refactor -(~2022.06, below the floor), so no compat shim is needed. +(2024.2.0 → 2026.4.0) — it post-dates the explicit-indexes refactor (~2022.06), so +no compat shim is needed. ## Feasibility matrix -Two axes — **feasible** (can flat+aux build an *equivalent model*?) and -**desirable** (our *opinion*: is it at least as good — *nicer* / *safer* / -*flexible*?): +Two axes, encoded separately so a glyph never means two things at once: **feasible** +— does flat+aux build an equivalent model? — and **desirable** — our opinion, is it +at least as good? -| bubble | feasible | desirable | -|---|---|---| -| 🟢 | works today (tested) | better | -| 🔵 | achievable (unimplemented) | — | -| ⚪ | — | parity | -| 🔴 | not feasible | worse | +- **feasible** (glyph): ✅ tested · 🔲 achievable, untested · ❌ no +- **desirable** (word): better · parity · worse -The 🟢 rows are tracked in `test/test_mi_feasibility.py` (which also solves a full -LP both ways, `test_per_period_lp_equivalent`, so the per-op rewrites are shown to -compose); PyPSA links are pinned at **v1.2.4** (commit [`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). - -All seven rows are the one `snapshot` `(period, timestep)` MI as it lives inside -linopy — in at `entry`, through the ops, out at `output`, parked on `snapshots -param` — and all are 🟢 (tested under both semantics). +All seven rows are the one `snapshot` MI inside linopy, and **all ✅** — tested under +both `legacy` and `v1`, with the build-time rewrites also shown to compose into an +identical LP (`test_per_period_lp_equivalent`). PyPSA links pinned at **v1.2.4** +([`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). | op | MI form | flat+aux form | feasible | desirable | PyPSA call site @ v1.2.4 | |---|---|---|---|---|---| -| **entry** | `coords=[mi]` | `reset_index(dim)` | 🟢 | 🟢 deletes MI machinery | [`constraints.py` L1052](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1052) (`from_pandas_multiindex`) | -| **level select** | `sel(snapshot=(p, slice))` | `where(period == p)` | 🟢 | ⚪ parity | [`constraints.py` L1235‑1248](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1235-L1248) (KVL; the per-period loop is topology-driven — `cycle_matrix(period)` — not MI, so flat+aux only swaps the `sns.get_loc` slice for `where`, the loop stays) | -| **period roll** | `roll(1)` + `_period_start_mask` | `groupby("period").map(roll)` | 🟢 (#751) | 🟢 mask-free (boundary from grouping) | [`constraints.py` L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694) (SOC) | -| **level groupby** | `groupby(MI level)` ❌ broken ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | `groupby("period").sum()` | 🟢 (#751) | 🟢 not just *nicer* — MI is *broken*, not merely workaround-y; flat+aux groups by the aux coord and just works (#751) | — | -| **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`, then period-start: wrap (cyclic) · `.where` drops the term (non-cyclic) · `mask=` drops the row (ramp) | 🟢 | 🟢 deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | -| **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | 🟢 | 🟢 cheap boundary conversion (PyPSA's choice) | — | -| **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | 🟢 | 🟢 removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | - -**No row needs a linopy change to be feasible.** The `entry` conversion is a -user-side `reset_index` with today's linopy; linopy auto-accepting `coords=[mi]` -and decomposing it would be *optional* input sugar. Dropping MI is a simplification -linopy chooses, not a capability it must add. - -**`level groupby` is the one row where flat+aux is *necessary*, not merely -preferable.** Elsewhere flat+aux is parity or deletes a workaround for something MI -still does; here grouping by an MI *level* is broken upstream (xarray#6836, outside -linopy's control) — there is no working native path at all. flat+aux groups by the -aux-coord *name*, which #751 routed onto the existing fast path. (The dense-`_term` -memory cost of groupby-sum, [#756](https://github.com/PyPSA/linopy/issues/756)/[#757](https://github.com/PyPSA/linopy/issues/757), -is real but representation-agnostic — both forms hit the same `unstack` — so it is -*not* what separates MI from flat+aux here.) - -The per-op cells above are the *nicer* facet. Two more facets are -**representation-wide**, not per-op (both verified this session): - -- **safer** — under v1 a conflicting level/aux coord *raises* (the general - aux-coord-conflict rule, `linopy/semantics.py: enforce_aux_conflict`); legacy - silently keeps one side ([#295](https://github.com/PyPSA/linopy/issues/295)). - MI "avoids" the conflict only by locking the levels into the index. -- **flexible** — flat+aux level coords can be `drop_vars`/`rename`/`assign_coords`'d - like any coord; on an MI the same reassignment raises *"cannot drop or update - coordinate … would corrupt the index"*. Removing MI removes that rigidity. - -The two axes tell different stories: flat+aux is **feasible almost everywhere** -and **desirable for every build-time op** — nicer (deletes MI machinery and -PyPSA's `.data`/`FILL_VALUE` workarounds), safer, and more flexible. The output -boundary is a **cheap conversion PyPSA owns** (`output` row, tested), and the one MI that lived *inside* a linopy object — `model.parameters.snapshots` (`snapshots param` row) — goes flat too. So the -conclusion for linopy is clean: **accept MI as input sugar, decompose on entry -(`reset_index`), and never reconstruct — flat in, flat out. linopy drops -MultiIndex from its mental model entirely.** +| **entry** | `coords=[mi]` | `reset_index(dim)` | ✅ | better — deletes MI machinery | [`constraints.py` L1052](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1052) (`from_pandas_multiindex`) | +| **level select** | `sel(snapshot=(p, slice))` | `where(period == p)` | ✅ | parity | [`constraints.py` L1235‑1248](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1235-L1248) (KVL; the per-period loop is topology-driven — `cycle_matrix(period)`, not MI — so flat+aux only swaps the `sns.get_loc` slice for `where`, the loop stays) | +| **period roll** | `roll(1)` + `_period_start_mask` | `groupby("period").map(roll)` | ✅ (#751) | better — mask-free (boundary from grouping) | [`constraints.py` L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694) (SOC) | +| **level groupby** | `groupby(MI level)` ❌ broken ([xarray#6836](https://github.com/pydata/xarray/issues/6836)) | `groupby("period").sum()` | ✅ (#751) | better — **necessary**: MI is *broken*, not just workaround-y; flat+aux groups by the aux coord and works (#751) | — | +| **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`, then period-start: wrap (cyclic) · `.where` term (non-cyclic) · `mask=` row (ramp) | ✅ | better — deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | +| **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | ✅ | better — cheap boundary conversion (PyPSA's choice) | — | +| **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | ✅ | better — removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | + +**No row needs a linopy change.** `entry` is a user-side `reset_index` with today's +linopy; auto-accepting `coords=[mi]` would be *optional* input sugar. Dropping MI is +a simplification linopy chooses, not a capability it must add. + +**`level groupby` is the one *necessary* row, not merely nicer.** Grouping by an MI +*level* is broken upstream (xarray#6836, outside linopy's control) — no working +native path — while flat+aux groups by the aux-coord name, which #751 put on the +fast path. (The dense-`_term` memory cost of groupby-sum, +[#756](https://github.com/PyPSA/linopy/issues/756)/[#757](https://github.com/PyPSA/linopy/issues/757), +is representation-agnostic — both forms hit the same `unstack` — so it is *not* what +separates them.) + +Beyond the per-op *nicer* facet, two are **representation-wide**: + +- **safer** — under v1 a conflicting level/aux coord *raises* (`enforce_aux_conflict`); + legacy silently keeps one side ([#295](https://github.com/PyPSA/linopy/issues/295)). + MI "avoids" the conflict only by locking levels into the index. +- **flexible** — flat+aux level coords `drop_vars`/`rename`/`assign_coords` like any + coord; on an MI the same raises *"would corrupt the index"*. ## Observed PyPSA MultiIndex usages — not linopy's to solve -Two further MultiIndex usages show up in PyPSA but **do not need solving inside -linopy** — they're cheap for PyPSA to handle at its own boundary. Each fails the -*in-linopy MI* test, on **opposite** axes: `stochastic` is inside linopy but isn't -an MI; `n.snapshots` is an MI but isn't inside linopy. +Two MultiIndex usages appear in PyPSA but **need no in-linopy solution** — cheap for +PyPSA to handle at its boundary. They fail the *in-linopy MI* test on **opposite** +axes: `stochastic` is inside linopy but isn't an MI; `n.snapshots` is an MI but isn't +inside linopy. | PyPSA MI usage | inside linopy? | an MI? | why linopy needn't solve it | call site @ v1.2.4 | |---|---|---|---|---| -| **stochastic** `(scenario, name)` | **yes** — `scenario` is a real dim (N-D) | **no** — only an `xarray.stack` at the pandas output | already the flat+aux shape in the model; the MI is output-cosmetic, rebuilt at the boundary like `output`. Watch only whether [#1484](https://github.com/PyPSA/PyPSA/issues/1484) ever makes it an *in-model* stacked index (v1.2.4 does not) | `(scenario,name)` pandas cols [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); output `.stack` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64); [#1154](https://github.com/PyPSA/PyPSA/pull/1154) | -| **n.snapshots** | **no** — a PyPSA `Network` attribute | **yes** — but the MI lives in PyPSA | linopy only `reindex_like`s against it; the MI never enters the model. Keep MI (wrap at its boundary) or flatten — PyPSA's call, decoupled from this decision | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | - -So the remaining cost is **not inside linopy**, and it is essentially **one** thing: -whether PyPSA keeps `n.snapshots` as MI — a cheap boundary wrap on its own side. +| **stochastic** `(scenario, name)` | **yes** — `scenario` is a real dim (N-D) | **no** — only an `xarray.stack` at the pandas output | already the flat+aux shape in the model; the MI is output-cosmetic, rebuilt at the boundary like `output`. Watch only whether [#1484](https://github.com/PyPSA/PyPSA/issues/1484) ever makes it an *in-model* stacked index (v1.2.4 does not) | pandas cols [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); output `.stack` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64); [#1154](https://github.com/PyPSA/PyPSA/pull/1154) | +| **n.snapshots** | **no** — a PyPSA `Network` attribute | **yes** — but the MI lives in PyPSA | linopy only `reindex_like`s against it; the MI never enters the model. Keep MI (wrap at its boundary) or flatten — PyPSA's call | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | **Side finding (not a row):** `Variable.sel` can't MI-tuple-select (`x.sel(snapshot=(p, slice))` → `InvalidIndexError`), which is why PyPSA drops to `.data` ([#752](https://github.com/PyPSA/linopy/issues/752) §2). Under flat+aux it -becomes `where(period == p)` / `isel`, removing the internals reach. Pinned by +becomes `where(period == p)` / `isel`. Pinned by `test_variable_mi_tuple_sel_not_forwarded`. -The matrix (all 🟢) answers the **steady-state (linopy)** question — tested under -both semantics. The **transition** is PyPSA-side: the two usages in the table -above, verified by solution-equivalence on real networks (multi-period, stochastic) -plus scoping the public `n.snapshots` change. - ## What `reset_index` changes (and doesn't) Today `snapshot` is a **single dimension** whose *index* is a `MultiIndex(period, @@ -149,62 +119,42 @@ direct `groupby`/`where`. That one line *is* the whole flat+aux transformation. ## Sub-decision: the snapshot dim coordinate -After `reset_index`, the dim is **coordinate-less** (xarray virtualizes `0..N-1` -on access). `timestep` can't be the coord (non-unique across periods); the unique -`(period, timestep)` pair *is* an MI. So: +After `reset_index` the dim is **coordinate-less** (xarray virtualizes `0..N-1`). +`timestep` can't be the coord (non-unique across periods); the unique `(period, +timestep)` pair *is* an MI. The choice is only positional vs label alignment: | dim coordinate | alignment | `.sel(snapshot=int)` | `.sel` by level tuple | |---|---|---|---| -| pure `reset_index` (none, virtual `0..N-1`) | positional | ✅ positional fallback | ❌ → `where(period==…)` | -| `+ assign_coords(RangeIndex)` (stored int index) | by label (`==` position here) | ✅ label lookup | ❌ → `where(period==…)` | - -Integer `.sel(snapshot=k)` works **either way** — on the coordinate-less dim -xarray degrades to positional selection (the tell: out-of-range raises -`IndexError`, not a label `KeyError`); with a `RangeIndex` it's a label lookup. -Since the coord is `0..N-1`, label `==` position, so they're indistinguishable for -valid integers. What flat+aux genuinely loses is `.sel(snapshot=(2020, "t1"))` — -selection by the **level tuple** (needs the MI) — replaced by `where(period==…)`; -that capability is gone under *both* flat variants, so it is not what the -coordinate choice decides. - -The only thing the coordinate choice decides is **alignment**: positional -(coordinate-less) vs label (`RangeIndex`). For snapshots this is a distinction -without a difference — one canonical `n.snapshots` order means positional and -label alignment coincide — and it matches the model linopy already uses for a -plain single-period datetime snapshot dim. The single line for §11 is just: -**snapshot alignment is positional, not tuple-identity.** +| pure `reset_index` (virtual `0..N-1`) | positional | ✅ positional fallback | ❌ → `where(period==…)` | +| `+ assign_coords(RangeIndex)` | by label (`==` position) | ✅ label lookup | ❌ → `where(period==…)` | + +Integer `.sel` works either way (label `==` position for `0..N-1`). What flat+aux +loses is selection by the **level tuple** `.sel(snapshot=(2020, "t1"))` — gone under +*both* variants, so it is not what the coordinate decides. With one canonical +`n.snapshots` order, positional and label alignment coincide, matching what linopy +already does for a plain datetime snapshot. The single rule for §11: **snapshot +alignment is positional, not tuple-identity.** ## Transition shape (PyPSA) -[#752](https://github.com/PyPSA/linopy/issues/752) catalogues PyPSA **reaching -into linopy internals** (term-storage: `.data`, `vars`/`coeffs`/`const`, `_term`, -the `FILL_VALUE` sentinel) — *not* MI per se. But several of those reaches are -**MI-driven workarounds**: the SOC `.data.sel().roll` + `FILL_VALUE` rebuild and -the growth `reindex_like(lhs.data)` linked above exist because MI groupby is broken -(the code comment says *"internal xarray multi-index difficulties"*, cf. -pydata/xarray#6836). #751 fixed the level groupby, so flat+aux **deletes** those -specific reaches rather than re-spelling them. These build-time sites `.sel` the -*stored* MI mid-build, so they migrate whether or not PyPSA re-stacks the output. -What's left to decide is only the **public `n.snapshots`** question (`n.snapshots` -row). Stochastic is *not* a second in-model MI — `scenario`/`name` are separate -N-D dims, the MI is pandas-output only — so the only watch item is whether -[PyPSA#1484](https://github.com/PyPSA/PyPSA/issues/1484) (Monte-Carlo) ever turns -that into an in-model stacked index; v1.2.4 does not. - -## Decision record (to fill once the open rows close) - -> _The `n.snapshots` scope (the one real open item) and the evidence rows that +[#752](https://github.com/PyPSA/linopy/issues/752) catalogues PyPSA **reaching into +linopy internals** (`.data`, `_term`, the `FILL_VALUE` sentinel) — *not* MI per se. +But several reaches are **MI-driven workarounds** (SOC `.data.sel().roll` + +`FILL_VALUE`, growth `reindex_like(lhs.data)`) that exist because MI groupby is broken +(code comment: *"internal xarray multi-index difficulties"*, xarray#6836). #751 fixed +the level groupby, so flat+aux **deletes** those reaches rather than re-spelling them +— and they `.sel` the *stored* MI mid-build, so they migrate regardless of how PyPSA +presents the output. + +## Decision record (to fill once the open item closes) + +> _The `n.snapshots` scope — the one real open item — and the evidence rows that > justify it. This note, once complete, is what closes #744._ - **linopy drops MultiIndex from its mental model** — accept MI as input sugar, - decompose on entry (`reset_index`), never reconstruct (flat in, flat out). - Internal normalization is safe to adopt now: cheap, canonical, version-safe - (`entry` row + version sweep). **[provisional — internals verified]** -- **Output: linopy returns flat** — the re-stack is a cheap boundary conversion - PyPSA owns (`output` row, tested); no MI adapter lives inside linopy. -- **`n.snapshots`** — PyPSA's independent, decoupled choice: keep MI (wrap at its - own boundary) or flatten. _TBD, PyPSA-side._ -- **Stochastic is N-D, not a second MI** (`stochastic` row) — `scenario`/`name` are - separate dims; the `(scenario, name)` MI is only an output `.stack`. No in-model MI - to remove. Watch only whether [#1484](https://github.com/PyPSA/PyPSA/issues/1484) - (Monte-Carlo) introduces an in-model stacked index; v1.2.4 does not. _Low risk._ + decompose on entry (`reset_index`), never reconstruct (flat in, flat out). Safe to + adopt now: cheap, canonical, version-safe. **[provisional — internals verified]** +- **Output: linopy returns flat** — the re-stack is a cheap boundary conversion PyPSA + owns (`output` row, tested); no MI adapter lives inside linopy. +- **`n.snapshots`** — PyPSA's independent, decoupled choice: keep MI (wrap at its own + boundary) or flatten. _TBD, PyPSA-side._ From 392b9ade0f34bc20d6197a16cfc81cf699058d3b Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 19:10:56 +0200 Subject: [PATCH 16/33] =?UTF-8?q?docs(arithmetics):=20correct=20"zero=20ch?= =?UTF-8?q?anges"=20=E2=86=92=20net=20subtraction;=20add=20strip=20analysi?= =?UTF-8?q?s=20(#744)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Zero linopy changes" overclaimed. Feasibility needs no new capability, but adopting flat+aux is a net simplification, not a no-op: a few small additive things (positional snapshot alignment; shrink MI-input to accept-then-reset_index) against a large deletion of first-class-MI machinery. Add a "What linopy can strip" section quantifying the deletion from a read-only code sweep: ~300 lines, ~half of it one cluster -- the alignment.py level-projection subsystem (_project_onto_multiindex_levels / _enforce_implicit_projections / _LevelProjection + the projections plumbing through broadcast_to_coords), which exists solely to make a single MI level align like a dimension and is dead under flat+aux; plus ~50 lines of netcdf MI flatten/reconstruct. Notes what stays (assign_multiindex_safe, internal _term/groupby stacking, semantics §11 aux-conflict). Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 45 +++++++++++++++++--- 1 file changed, 40 insertions(+), 5 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index f135a6ae..ee9d3158 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -8,11 +8,17 @@ ## Verdict -**Feasible — yes, with zero linopy changes.** linopy accepts an MI snapshot as -*input sugar*, `reset_index` on entry, and goes **flat in / flat out** — never -reconstructing it. PyPSA uses a MultiIndex on exactly one axis, `snapshot` -`(period, timestep)`, and all seven of its in-linopy uses have a flat+aux form that -builds the identical model, each tested under both semantics. +**Feasible — and the change is mostly *subtraction*.** No new capability is +required: the flat+aux model builds with today's linopy, and all seven in-linopy MI +uses have a flat+aux form that builds the identical model, tested under both +semantics. PyPSA uses a MultiIndex on exactly one axis, `snapshot` +`(period, timestep)`, accepted as *input sugar*, `reset_index`'d on entry — +**flat in / flat out**, never reconstructed. + +Adopting it as v1 is a **net simplification**: a few small things to get right +(positional snapshot alignment; optionally auto-normalizing MI input), against a lot +of first-class-MI machinery linopy gets to **delete** (see *What linopy can strip* +below). So it is not "zero changes" — it is mostly deletions. **One open item, and it is PyPSA's, not linopy's:** whether `n.snapshots` stays a MultiIndex — a cheap boundary wrap PyPSA owns. @@ -87,6 +93,35 @@ Beyond the per-op *nicer* facet, two are **representation-wide**: - **flexible** — flat+aux level coords `drop_vars`/`rename`/`assign_coords` like any coord; on an MI the same raises *"would corrupt the index"*. +## What linopy can strip + +Adopting flat+aux is a **net deletion** of first-class-MI machinery — roughly +**~300 lines**, and concentrated, not diffuse. About **half is one cluster**: the +`alignment.py` *level-projection* subsystem, whose only job is to make an operand +indexed by a single MI *level* align against the full MI dim (levels behaving like +pseudo-dimensions). Under flat+aux the levels simply *are* aux coords, so the whole +subsystem is dead code. + +| strip | what it does today | ~lines | +|---|---|---| +| **`alignment.py` level-projection** — `_project_onto_multiindex_levels`, `_enforce_implicit_projections`, `_LevelProjection`, the `projections` plumbing through `broadcast_to_coords`, plus the MI branches in `_expand_missing_dims`/`validate_alignment` and `_as_multiindex` | align a single-level operand against a full MI dim | ~150 | +| **netcdf MI (de)serialization** — `io.py` flatten-on-write + reconstruct (`{dim}_multiindex` attr); `common.py` MI level/code (de)serialize | flatten MI to store, rebuild MI on read | ~50 (keep a read-only shim for old `.nc`) | +| **scattered MI guards/branches** in alignment & coords | skip-logic that only fires when a dim is an MI | remainder | + +**What stays** (so the claim is honest): `assign_multiindex_safe` and the internal +`_term`/`_factor`/groupby stacking are *general* helpers over linopy's own helper +indexes — unrelated to the snapshot MI. The `semantics.py` shared-dim reorder and +§11 aux-coord-conflict logic not only stay but become **more central** — aux coords +are the new home for the levels. + +**The "small things to get right"** are additive and minor: fix positional snapshot +alignment (§11 — already how a plain datetime snapshot behaves) and shrink the +MI-input path to *accept-then-`reset_index`* rather than store. A few lines added at +the boundary; ~300 removed from the core. + +> _Strip inventory from a read-only code sweep (Claude Code) of the v1 working tree; +> line counts are order-of-magnitude. Not yet executed — this scopes the deletion._ + ## Observed PyPSA MultiIndex usages — not linopy's to solve Two MultiIndex usages appear in PyPSA but **need no in-linopy solution** — cheap for From 976ab801e70ae7b7a3479dfeee8603153e118ace Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 19:44:05 +0200 Subject: [PATCH 17/33] docs(arithmetics): frame the verdict as four wins, not just code-strip (#744) The case for dropping first-class MI isn't only "~300 fewer lines". Reframe the Verdict around four axes, each backed by a section below: - simpler implementation (What linopy can strip) - simpler mental model (What reset_index changes) - enables features (level groupby works; removes the #752 internals coupling) - no UX cost (MI still accepted in / re-stacked out; only level-tuple .sel lost) Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 22 ++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index ee9d3158..6a4b83d5 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -15,10 +15,24 @@ semantics. PyPSA uses a MultiIndex on exactly one axis, `snapshot` `(period, timestep)`, accepted as *input sugar*, `reset_index`'d on entry — **flat in / flat out**, never reconstructed. -Adopting it as v1 is a **net simplification**: a few small things to get right -(positional snapshot alignment; optionally auto-normalizing MI input), against a lot -of first-class-MI machinery linopy gets to **delete** (see *What linopy can strip* -below). So it is not "zero changes" — it is mostly deletions. +Adopting it as v1 is not "zero changes", but it pays off on four axes: + +- **Simpler implementation** — deletes ~300 lines of first-class-MI machinery, half + of it one cluster (the `alignment.py` level-projection subsystem); see *What linopy + can strip*. +- **Simpler mental model** — one flat `snapshot` dim with `period`/`timestep` as + ordinary aux coords; no "levels behaving like dimensions" to reason about. + `reset_index` *is* the whole transform. +- **Enables features** — the MI-level groupby that's broken upstream (xarray#6836) + just works on a flat dim (#751); aux coords compose with `groupby`/`where`/ + `drop_vars` where an MI can't; and it removes the MI coupling that forces PyPSA to + reach into linopy internals ([#752](https://github.com/PyPSA/linopy/issues/752)). +- **No UX cost** — MI is still accepted as input and re-stacked on output if the + caller wants; the only thing lost is `.sel` by level *tuple*, replaced by + `where(period==…)`. + +The few small additive things to get right (positional snapshot alignment; shrink the +MI-input path to *accept-then-`reset_index`*) are minor against that. **One open item, and it is PyPSA's, not linopy's:** whether `n.snapshots` stays a MultiIndex — a cheap boundary wrap PyPSA owns. From 614804c57648f05e4a0619d9bcf4f14bf8a600e2 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 19:46:54 +0200 Subject: [PATCH 18/33] docs(arithmetics): frame the mental-model win as linopy's own (#744) The simpler mental model is linopy's: it stops having to know pd.MultiIndex exists or cover its quirks (level alignment, index-corruption guards, reconstruction) and becomes purely xarray-native -- plain dims + aux coords, what it's already built on. MI lives only at the boundary as input/output sugar, never inside linopy. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 6a4b83d5..493e9b82 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -20,9 +20,12 @@ Adopting it as v1 is not "zero changes", but it pays off on four axes: - **Simpler implementation** — deletes ~300 lines of first-class-MI machinery, half of it one cluster (the `alignment.py` level-projection subsystem); see *What linopy can strip*. -- **Simpler mental model** — one flat `snapshot` dim with `period`/`timestep` as - ordinary aux coords; no "levels behaving like dimensions" to reason about. - `reset_index` *is* the whole transform. +- **Simpler mental model — linopy's own.** linopy stops having to know + `pd.MultiIndex` exists at all: no MI quirks to cover (level alignment, *"would + corrupt the index"* guards, reconstruction), no levels masquerading as dimensions. + Its model becomes purely **xarray-native** — plain dims + ordinary aux coords, + exactly what linopy is already built on. MI lives only at the boundary as + input/output sugar, never inside linopy. - **Enables features** — the MI-level groupby that's broken upstream (xarray#6836) just works on a flat dim (#751); aux coords compose with `groupby`/`where`/ `drop_vars` where an MI can't; and it removes the MI coupling that forces PyPSA to From e834e486155657fd161dac34ab4ee26c2461b19a Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 19:50:52 +0200 Subject: [PATCH 19/33] docs(arithmetics): rewrite Verdict in the maintainer's voice (#744) Lead with the linopy maintainer's actual question -- can the library be simplified without destabilising PyPSA, its primary consumer -- and answer it: PyPSA's interface doesn't move (MI in, MI-indexed out as boundary sugar), every in-linopy use has a tested identical-model flat+aux form, so nothing PyPSA relies on breaks. Stability is the frame; the wins (less to maintain, xarray-native internals, unblocks work) follow from it, and the one open item is PyPSA's own decoupled n.snapshots call. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 58 +++++++++----------- 1 file changed, 27 insertions(+), 31 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 493e9b82..7cedfba3 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -8,37 +8,33 @@ ## Verdict -**Feasible — and the change is mostly *subtraction*.** No new capability is -required: the flat+aux model builds with today's linopy, and all seven in-linopy MI -uses have a flat+aux form that builds the identical model, tested under both -semantics. PyPSA uses a MultiIndex on exactly one axis, `snapshot` -`(period, timestep)`, accepted as *input sugar*, `reset_index`'d on entry — -**flat in / flat out**, never reconstructed. - -Adopting it as v1 is not "zero changes", but it pays off on four axes: - -- **Simpler implementation** — deletes ~300 lines of first-class-MI machinery, half - of it one cluster (the `alignment.py` level-projection subsystem); see *What linopy - can strip*. -- **Simpler mental model — linopy's own.** linopy stops having to know - `pd.MultiIndex` exists at all: no MI quirks to cover (level alignment, *"would - corrupt the index"* guards, reconstruction), no levels masquerading as dimensions. - Its model becomes purely **xarray-native** — plain dims + ordinary aux coords, - exactly what linopy is already built on. MI lives only at the boundary as - input/output sugar, never inside linopy. -- **Enables features** — the MI-level groupby that's broken upstream (xarray#6836) - just works on a flat dim (#751); aux coords compose with `groupby`/`where`/ - `drop_vars` where an MI can't; and it removes the MI coupling that forces PyPSA to - reach into linopy internals ([#752](https://github.com/PyPSA/linopy/issues/752)). -- **No UX cost** — MI is still accepted as input and re-stacked on output if the - caller wants; the only thing lost is `.sel` by level *tuple*, replaced by - `where(period==…)`. - -The few small additive things to get right (positional snapshot alignment; shrink the -MI-input path to *accept-then-`reset_index`*) are minor against that. - -**One open item, and it is PyPSA's, not linopy's:** whether `n.snapshots` stays a -MultiIndex — a cheap boundary wrap PyPSA owns. +**Feasible, and PyPSA stays stable.** The maintainer's question is whether linopy +can be simplified *without destabilising PyPSA*, its primary consumer — and it can. +The flat+aux model builds with today's linopy (no new capability), and every one of +the seven in-linopy MI uses has a flat+aux form that builds the **identical** model, +tested under both `legacy` and `v1`. PyPSA's interface does not move: it hands linopy +an MI snapshot and gets MI-indexed results back exactly as today. MI becomes +**boundary sugar** — `reset_index` on the way in, re-stack on the way out — so +nothing PyPSA relies on breaks. + +With stability assured, the change is mostly *subtraction*, and it pays off where a +maintainer feels it: + +- **Less to maintain** — ~300 lines of first-class-MI machinery go, half of it one + cluster (the `alignment.py` level-projection subsystem); see *What linopy can strip*. +- **xarray-native internals** — linopy stops having to know `pd.MultiIndex` exists or + cover its quirks (level alignment, *"would corrupt the index"* guards, + reconstruction); the model is just dims + ordinary aux coords, what linopy is + already built on. +- **Unblocks work** — the MI-level groupby broken upstream (xarray#6836) works on a + flat dim (#751), and the MI coupling that makes PyPSA reach into linopy internals + ([#752](https://github.com/PyPSA/linopy/issues/752)) goes away. + +The small additive work (positional snapshot alignment; shrink the MI-input path to +*accept-then-`reset_index`*) is minor against that. + +**One open item, and it is PyPSA's own, decoupled call:** whether `n.snapshots` stays +a MultiIndex — a cheap boundary wrap. linopy works either way. Everything below is evidence: the **matrix** of the seven in-linopy ops (settled, tested now), then the two **PyPSA-side usages** (the transition, PyPSA-owned). From f0a02d35121797b287a38304db6a4ff9d5b8bde6 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 19:53:40 +0200 Subject: [PATCH 20/33] docs(arithmetics): attribute cases to PyPSA; state the generalization assumption (#744) The rows are PyPSA's MI uses *of* linopy, not linopy's own -- reword accordingly. And make the epistemics explicit: PyPSA is the one consumer audited, so these are a proof set, not a proof of universality. The argument still generalises (after reset_index everything is ordinary xarray linopy already handles; the only MI-specific capability lost for any user is level-tuple .sel), but we now state the assumption that no other user has an infeasible MI case, and invite counterexamples. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 21 +++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 7cedfba3..392bb31b 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -11,8 +11,8 @@ **Feasible, and PyPSA stays stable.** The maintainer's question is whether linopy can be simplified *without destabilising PyPSA*, its primary consumer — and it can. The flat+aux model builds with today's linopy (no new capability), and every one of -the seven in-linopy MI uses has a flat+aux form that builds the **identical** model, -tested under both `legacy` and `v1`. PyPSA's interface does not move: it hands linopy +the **seven MI uses PyPSA makes of linopy's model** has a flat+aux form that builds +the **identical** model, tested under both `legacy` and `v1`. PyPSA's interface does not move: it hands linopy an MI snapshot and gets MI-indexed results back exactly as today. MI becomes **boundary sugar** — `reset_index` on the way in, re-stack on the way out — so nothing PyPSA relies on breaks. @@ -36,8 +36,14 @@ The small additive work (positional snapshot alignment; shrink the MI-input path **One open item, and it is PyPSA's own, decoupled call:** whether `n.snapshots` stays a MultiIndex — a cheap boundary wrap. linopy works either way. -Everything below is evidence: the **matrix** of the seven in-linopy ops (settled, -tested now), then the two **PyPSA-side usages** (the transition, PyPSA-owned). +Everything below is evidence. The cases are **PyPSA's observed MI uses of linopy**, +split by whether they enter linopy's model — the **matrix** (seven in-model uses, +each with a tested flat+aux equivalent), then the two **boundary usages** (PyPSA-side). +They are a proof set, not a proof of universality: PyPSA is the one consumer we +audited. But the argument generalises — once `reset_index` runs (general to *any* MI) +the data is ordinary xarray that linopy already handles on any dim, so the only +MI-specific capability lost for *any* user is `.sel` by level *tuple* (→ `where`). We +assume no linopy user needs that as such; counterexamples welcome. ## Scope: one axis, `snapshot` @@ -71,9 +77,10 @@ at least as good? - **feasible** (glyph): ✅ tested · 🔲 achievable, untested · ❌ no - **desirable** (word): better · parity · worse -All seven rows are the one `snapshot` MI inside linopy, and **all ✅** — tested under -both `legacy` and `v1`, with the build-time rewrites also shown to compose into an -identical LP (`test_per_period_lp_equivalent`). PyPSA links pinned at **v1.2.4** +All seven rows are PyPSA's uses of the one `snapshot` MI inside linopy's model, and +**all ✅** — tested under both `legacy` and `v1`, with the build-time rewrites also +shown to compose into an identical LP (`test_per_period_lp_equivalent`). PyPSA links +pinned at **v1.2.4** ([`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). | op | MI form | flat+aux form | feasible | desirable | PyPSA call site @ v1.2.4 | From 93025955c61885ea887fb024ea7c3259de9996e2 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 20:00:31 +0200 Subject: [PATCH 21/33] =?UTF-8?q?docs(arithmetics):=20dense=20rewrite=20?= =?UTF-8?q?=E2=80=94=20Decision=20+=20Appendix=20shape=20(#744)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructure to one decision screen + appendix (the chosen shape), 216 → 141 lines: - Verdict: claim + 3 wins + open item + proof-set caveat, tightened. - The evidence: both tables (in-model matrix, boundary usages) under one heading; the three post-matrix findings paragraphs collapse to one caption; Scope and the side-finding fold in. - The payoff: strip table + a line; "stays"/"additive" compressed. - Appendix: the three overlapping mechanics sections merged/condensed (reset_index + data-model table, dim-coordinate sub-decision, transition shape) + decision record. Each claim stated once; explanatory prose turned into captions. Tables and all call-site links preserved verbatim; only the #756/#757 links in a memory-cost aside were trimmed (the point survives without them). Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 264 +++++++------------ 1 file changed, 95 insertions(+), 169 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 392bb31b..a472078e 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -1,87 +1,42 @@ # MultiIndex feasibility for v1 (#744) -> Verification note (Claude Code, prompted by @FBumann), 2026-06-29. Tracks whether -> linopy can drop **first-class `pd.MultiIndex`** support in v1 for a **flat dim + -> auxiliary level coords** model. Home: [#744](https://github.com/PyPSA/linopy/issues/744). -> Equality checks: [`test/test_mi_feasibility.py`](../test/test_mi_feasibility.py) -> (runs under both `legacy` and `v1`). +> Verification note (Claude Code, prompted by @FBumann), 2026-06-29. Can linopy v1 +> drop first-class **`pd.MultiIndex`** for a **flat dim + aux level coords** model? +> Discussion: [#744](https://github.com/PyPSA/linopy/issues/744). Equality checks: +> [`test/test_mi_feasibility.py`](../test/test_mi_feasibility.py) (both `legacy` and +> `v1`). PyPSA refs pinned at **v1.2.4** ([`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). ## Verdict -**Feasible, and PyPSA stays stable.** The maintainer's question is whether linopy -can be simplified *without destabilising PyPSA*, its primary consumer — and it can. -The flat+aux model builds with today's linopy (no new capability), and every one of -the **seven MI uses PyPSA makes of linopy's model** has a flat+aux form that builds -the **identical** model, tested under both `legacy` and `v1`. PyPSA's interface does not move: it hands linopy -an MI snapshot and gets MI-indexed results back exactly as today. MI becomes -**boundary sugar** — `reset_index` on the way in, re-stack on the way out — so -nothing PyPSA relies on breaks. - -With stability assured, the change is mostly *subtraction*, and it pays off where a -maintainer feels it: - -- **Less to maintain** — ~300 lines of first-class-MI machinery go, half of it one - cluster (the `alignment.py` level-projection subsystem); see *What linopy can strip*. -- **xarray-native internals** — linopy stops having to know `pd.MultiIndex` exists or - cover its quirks (level alignment, *"would corrupt the index"* guards, - reconstruction); the model is just dims + ordinary aux coords, what linopy is - already built on. -- **Unblocks work** — the MI-level groupby broken upstream (xarray#6836) works on a - flat dim (#751), and the MI coupling that makes PyPSA reach into linopy internals - ([#752](https://github.com/PyPSA/linopy/issues/752)) goes away. - -The small additive work (positional snapshot alignment; shrink the MI-input path to -*accept-then-`reset_index`*) is minor against that. - -**One open item, and it is PyPSA's own, decoupled call:** whether `n.snapshots` stays -a MultiIndex — a cheap boundary wrap. linopy works either way. - -Everything below is evidence. The cases are **PyPSA's observed MI uses of linopy**, -split by whether they enter linopy's model — the **matrix** (seven in-model uses, -each with a tested flat+aux equivalent), then the two **boundary usages** (PyPSA-side). -They are a proof set, not a proof of universality: PyPSA is the one consumer we -audited. But the argument generalises — once `reset_index` runs (general to *any* MI) -the data is ordinary xarray that linopy already handles on any dim, so the only -MI-specific capability lost for *any* user is `.sel` by level *tuple* (→ `where`). We -assume no linopy user needs that as such; counterexamples welcome. - -## Scope: one axis, `snapshot` - -Inside the linopy model PyPSA uses a `pd.MultiIndex` only on the `snapshot` -dimension; it rides that axis through the whole lifecycle — in at `entry`, through -the per-period ops, out at `output` (`solution`/`dual` carry `snapshot`, so they are -MI-indexed too), parked on `snapshots param`. Two usages merely *look* MultiIndex-ish -and are handled PyPSA-side (their own table below): `stochastic` (`scenario`/`name` -are separate N-D dims; the MI is only an `xarray.stack` at the pandas output) and -`n.snapshots` (a real MI, but PyPSA's *public API*, never a linopy model index). - -## Data model under test - -| | today (MI) | proposed (flat+aux) | -|---|---|---| -| snapshot dim | `MultiIndex[(period, timestep)]` | flat `snapshot` dim | -| level identity | MI levels | `period`/`timestep` **aux coords** on `snapshot` | -| entry conversion | — | `obj.reset_index("snapshot")` (canonical xarray, no custom logic) | -| alignment | tuple-identity | **positional** (one canonical snapshot order) | +**Feasible, and PyPSA stays stable.** Can linopy be simplified without destabilising +PyPSA, its primary consumer? Yes. PyPSA's interface does not move — it hands linopy an +MI `snapshot` `(period, timestep)` and gets MI-indexed results back; MI is **boundary +sugar** (`reset_index` in, re-stack out), never inside the model. All **seven ways +PyPSA puts that MI into linopy's model** have a flat+aux form building the *identical* +model, tested under both semantics. The change is mostly **subtraction**: -The entry conversion is byte-identical across linopy's supported xarray range -(2024.2.0 → 2026.4.0) — it post-dates the explicit-indexes refactor (~2022.06), so -no compat shim is needed. +- **less to maintain** — ~300 lines of first-class-MI machinery deleted, half one + cluster (see *The payoff*). +- **xarray-native internals** — linopy stops knowing `pd.MultiIndex` exists or covering + its quirks; the model is just dims + ordinary aux coords. +- **unblocks work** — the MI-level groupby broken upstream (xarray#6836) works flat + (#751); the MI coupling forcing PyPSA into linopy internals ([#752](https://github.com/PyPSA/linopy/issues/752)) goes. -## Feasibility matrix +Small additive work: positional snapshot alignment; shrink MI-input to +accept-then-`reset_index`. **One open item, PyPSA's own decoupled call:** whether +`n.snapshots` stays an MI (a cheap boundary wrap) — linopy works either way. -Two axes, encoded separately so a glyph never means two things at once: **feasible** -— does flat+aux build an equivalent model? — and **desirable** — our opinion, is it -at least as good? +*Proof set, not universal: PyPSA is the one consumer audited. But after `reset_index` +(general to any MI) everything is ordinary xarray, so the only MI-specific capability +lost for any user is `.sel` by level tuple (→ `where`). Counterexamples welcome.* -- **feasible** (glyph): ✅ tested · 🔲 achievable, untested · ❌ no -- **desirable** (word): better · parity · worse +## The evidence -All seven rows are PyPSA's uses of the one `snapshot` MI inside linopy's model, and -**all ✅** — tested under both `legacy` and `v1`, with the build-time rewrites also -shown to compose into an identical LP (`test_per_period_lp_equivalent`). PyPSA links -pinned at **v1.2.4** -([`fb425cb`](https://github.com/PyPSA/PyPSA/tree/v1.2.4)). +PyPSA's observed MI uses of linopy, split by whether the MI **enters the model**. +In-model uses are the matrix — all ✅ (tested under `legacy`+`v1`; build-time rewrites +also compose into an identical LP, `test_per_period_lp_equivalent`). Glyph = +**feasible** (✅ tested · 🔲 achievable · ❌ no); word = **desirable** (better · parity +· worse). | op | MI form | flat+aux form | feasible | desirable | PyPSA call site @ v1.2.4 | |---|---|---|---|---|---| @@ -93,34 +48,32 @@ pinned at **v1.2.4** | **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | ✅ | better — cheap boundary conversion (PyPSA's choice) | — | | **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | ✅ | better — removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | -**No row needs a linopy change.** `entry` is a user-side `reset_index` with today's -linopy; auto-accepting `coords=[mi]` would be *optional* input sugar. Dropping MI is -a simplification linopy chooses, not a capability it must add. +No row needs a linopy change (`entry` is a user-side `reset_index`). `level groupby` +is the one *necessary* row — an MI level can't be grouped (broken upstream, +xarray#6836; the dense-`_term` cost is representation-agnostic, not the differentiator), +flat+aux just works (#751); the rest are nicer or parity. Representation-wide, flat+aux +is also **safer** (v1 *raises* on a conflicting aux coord via `enforce_aux_conflict`; +MI only hides it, [#295](https://github.com/PyPSA/linopy/issues/295)) and **flexible** +(level coords `drop_vars`/`rename` freely; an MI raises *"would corrupt the index"*). -**`level groupby` is the one *necessary* row, not merely nicer.** Grouping by an MI -*level* is broken upstream (xarray#6836, outside linopy's control) — no working -native path — while flat+aux groups by the aux-coord name, which #751 put on the -fast path. (The dense-`_term` memory cost of groupby-sum, -[#756](https://github.com/PyPSA/linopy/issues/756)/[#757](https://github.com/PyPSA/linopy/issues/757), -is representation-agnostic — both forms hit the same `unstack` — so it is *not* what -separates them.) +**Boundary uses** — two MI usages need no in-linopy solution, failing the in-model test +on opposite axes: `stochastic` is inside linopy but not an MI; `n.snapshots` is an MI +but not inside linopy. -Beyond the per-op *nicer* facet, two are **representation-wide**: +| PyPSA MI usage | inside linopy? | an MI? | why linopy needn't solve it | call site @ v1.2.4 | +|---|---|---|---|---| +| **stochastic** `(scenario, name)` | **yes** — `scenario` is a real dim (N-D) | **no** — only an `xarray.stack` at the pandas output | already the flat+aux shape in the model; the MI is output-cosmetic, rebuilt at the boundary like `output`. Watch only whether [#1484](https://github.com/PyPSA/PyPSA/issues/1484) ever makes it an *in-model* stacked index (v1.2.4 does not) | pandas cols [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); output `.stack` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64); [#1154](https://github.com/PyPSA/PyPSA/pull/1154) | +| **n.snapshots** | **no** — a PyPSA `Network` attribute | **yes** — but the MI lives in PyPSA | linopy only `reindex_like`s against it; the MI never enters the model. Keep MI (wrap at its boundary) or flatten — PyPSA's call | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | -- **safer** — under v1 a conflicting level/aux coord *raises* (`enforce_aux_conflict`); - legacy silently keeps one side ([#295](https://github.com/PyPSA/linopy/issues/295)). - MI "avoids" the conflict only by locking levels into the index. -- **flexible** — flat+aux level coords `drop_vars`/`rename`/`assign_coords` like any - coord; on an MI the same raises *"would corrupt the index"*. +(`Variable.sel` can't MI-tuple-select → `InvalidIndexError`, why PyPSA drops to `.data` +[#752](https://github.com/PyPSA/linopy/issues/752) §2; flat+aux makes it `where`/`isel`. +Pinned by `test_variable_mi_tuple_sel_not_forwarded`.) -## What linopy can strip +## The payoff -Adopting flat+aux is a **net deletion** of first-class-MI machinery — roughly -**~300 lines**, and concentrated, not diffuse. About **half is one cluster**: the -`alignment.py` *level-projection* subsystem, whose only job is to make an operand -indexed by a single MI *level* align against the full MI dim (levels behaving like -pseudo-dimensions). Under flat+aux the levels simply *are* aux coords, so the whole -subsystem is dead code. +Adopting flat+aux **deletes** ~300 lines of first-class-MI machinery — concentrated, +not diffuse: ~half is the `alignment.py` *level-projection* subsystem, whose only job +is to make a single MI level align like a dimension (dead once levels are aux coords). | strip | what it does today | ~lines | |---|---|---| @@ -128,88 +81,61 @@ subsystem is dead code. | **netcdf MI (de)serialization** — `io.py` flatten-on-write + reconstruct (`{dim}_multiindex` attr); `common.py` MI level/code (de)serialize | flatten MI to store, rebuild MI on read | ~50 (keep a read-only shim for old `.nc`) | | **scattered MI guards/branches** in alignment & coords | skip-logic that only fires when a dim is an MI | remainder | -**What stays** (so the claim is honest): `assign_multiindex_safe` and the internal -`_term`/`_factor`/groupby stacking are *general* helpers over linopy's own helper -indexes — unrelated to the snapshot MI. The `semantics.py` shared-dim reorder and -§11 aux-coord-conflict logic not only stay but become **more central** — aux coords -are the new home for the levels. +Stays: `assign_multiindex_safe` and internal `_term`/`_factor`/groupby stacking +(general helpers over linopy's own indexes, unrelated to snapshot-MI), and the §11 +aux-conflict logic — which gets *more* central, aux coords being the new home for the +levels. -**The "small things to get right"** are additive and minor: fix positional snapshot -alignment (§11 — already how a plain datetime snapshot behaves) and shrink the -MI-input path to *accept-then-`reset_index`* rather than store. A few lines added at -the boundary; ~300 removed from the core. +> _Strip from a read-only sweep of the v1 tree; counts order-of-magnitude, not yet +> executed._ -> _Strip inventory from a read-only code sweep (Claude Code) of the v1 working tree; -> line counts are order-of-magnitude. Not yet executed — this scopes the deletion._ +## Appendix -## Observed PyPSA MultiIndex usages — not linopy's to solve +### `reset_index` is the whole transform -Two MultiIndex usages appear in PyPSA but **need no in-linopy solution** — cheap for -PyPSA to handle at its boundary. They fail the *in-linopy MI* test on **opposite** -axes: `stochastic` is inside linopy but isn't an MI; `n.snapshots` is an MI but isn't -inside linopy. +`snapshot` is one dim whose *index* is a `MultiIndex(period, timestep)`; +`period`/`timestep` are **levels, never dims** — they become a real dim only when +`.sel(period=p)` collapses the MI (xarray renames `snapshot → timestep`), which is why +per-period code must `.sel`-collapse, loop, or reach into `.data`. `reset_index` flips +**only** the index type (MI → flat), not the dim count (`('snapshot','name')` before +and after), demoting levels to aux coords — collapse/loop/`.data` becomes direct +`groupby`/`where`. Canonical xarray, byte-identical across the supported range +(2024.2.0 → 2026.4.0; post-dates the ~2022.06 explicit-indexes refactor, no shim). -| PyPSA MI usage | inside linopy? | an MI? | why linopy needn't solve it | call site @ v1.2.4 | -|---|---|---|---|---| -| **stochastic** `(scenario, name)` | **yes** — `scenario` is a real dim (N-D) | **no** — only an `xarray.stack` at the pandas output | already the flat+aux shape in the model; the MI is output-cosmetic, rebuilt at the boundary like `output`. Watch only whether [#1484](https://github.com/PyPSA/PyPSA/issues/1484) ever makes it an *in-model* stacked index (v1.2.4 does not) | pandas cols [`common.py` L78‑80](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L78-L80); per-scenario loop [`optimize.py` L225‑229](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L225-L229); [`isel(scenario=0)` L1092‑1094](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1092-L1094); output `.stack` [`array.py` L55‑64](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/components/array.py#L55-L64); [#1154](https://github.com/PyPSA/PyPSA/pull/1154) | -| **n.snapshots** | **no** — a PyPSA `Network` attribute | **yes** — but the MI lives in PyPSA | linopy only `reindex_like`s against it; the MI never enters the model. Keep MI (wrap at its boundary) or flatten — PyPSA's call | [`global_constraints.py` L267](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/global_constraints.py#L267) (`reindex_like(lhs.data)`) | - -**Side finding (not a row):** `Variable.sel` can't MI-tuple-select -(`x.sel(snapshot=(p, slice))` → `InvalidIndexError`), which is why PyPSA drops to -`.data` ([#752](https://github.com/PyPSA/linopy/issues/752) §2). Under flat+aux it -becomes `where(period == p)` / `isel`. Pinned by -`test_variable_mi_tuple_sel_not_forwarded`. - -## What `reset_index` changes (and doesn't) - -Today `snapshot` is a **single dimension** whose *index* is a `MultiIndex(period, -timestep)`; `period`/`timestep` are **levels, never dimensions**. They surface as a -real dim only *momentarily* — `.sel(period=p)` collapses the 2-level MI to one level -and xarray renames `snapshot → timestep`; the two never coexist as independent dims. -That is exactly why per-period code must `.sel`-collapse, loop, or reach into `.data` -(the SOC/KVL/objective coupling). `reset_index("snapshot")` flips **only** the index -type (MultiIndex → flat), not the dim count — `('snapshot', 'name')` before and -after — demoting the levels to ordinary aux coords; collapse/loop/`.data` then becomes -direct `groupby`/`where`. That one line *is* the whole flat+aux transformation. +| | today (MI) | flat+aux | +|---|---|---| +| `snapshot` | one dim, index = `MultiIndex(period, timestep)` | one flat dim | +| levels | MI levels (must `.sel`/collapse to use) | `period`/`timestep` aux coords | +| per-period op | `.sel(period=p)` / loop / `.data` | `groupby("period")` / `where` | +| alignment | tuple-identity | positional (one canonical order) | -## Sub-decision: the snapshot dim coordinate +### Snapshot dim coordinate -After `reset_index` the dim is **coordinate-less** (xarray virtualizes `0..N-1`). -`timestep` can't be the coord (non-unique across periods); the unique `(period, -timestep)` pair *is* an MI. The choice is only positional vs label alignment: +After `reset_index` the dim is **coordinate-less** (`0..N-1` virtual). The only choice +is positional vs label alignment — both lose level-tuple `.sel`: -| dim coordinate | alignment | `.sel(snapshot=int)` | `.sel` by level tuple | +| dim coordinate | alignment | `.sel(int)` | `.sel(level tuple)` | |---|---|---|---| -| pure `reset_index` (virtual `0..N-1`) | positional | ✅ positional fallback | ❌ → `where(period==…)` | -| `+ assign_coords(RangeIndex)` | by label (`==` position) | ✅ label lookup | ❌ → `where(period==…)` | +| pure `reset_index` (virtual `0..N-1`) | positional | ✅ positional | ❌ → `where` | +| `+ assign_coords(RangeIndex)` | by label (`==` position) | ✅ label | ❌ → `where` | -Integer `.sel` works either way (label `==` position for `0..N-1`). What flat+aux -loses is selection by the **level tuple** `.sel(snapshot=(2020, "t1"))` — gone under -*both* variants, so it is not what the coordinate decides. With one canonical -`n.snapshots` order, positional and label alignment coincide, matching what linopy -already does for a plain datetime snapshot. The single rule for §11: **snapshot -alignment is positional, not tuple-identity.** +With one canonical `n.snapshots` order, positional `==` label, matching a plain +datetime snapshot. **§11 rule: snapshot alignment is positional, not tuple-identity.** -## Transition shape (PyPSA) +### Transition shape (PyPSA) [#752](https://github.com/PyPSA/linopy/issues/752) catalogues PyPSA **reaching into -linopy internals** (`.data`, `_term`, the `FILL_VALUE` sentinel) — *not* MI per se. -But several reaches are **MI-driven workarounds** (SOC `.data.sel().roll` + -`FILL_VALUE`, growth `reindex_like(lhs.data)`) that exist because MI groupby is broken -(code comment: *"internal xarray multi-index difficulties"*, xarray#6836). #751 fixed -the level groupby, so flat+aux **deletes** those reaches rather than re-spelling them -— and they `.sel` the *stored* MI mid-build, so they migrate regardless of how PyPSA -presents the output. - -## Decision record (to fill once the open item closes) - -> _The `n.snapshots` scope — the one real open item — and the evidence rows that -> justify it. This note, once complete, is what closes #744._ - -- **linopy drops MultiIndex from its mental model** — accept MI as input sugar, - decompose on entry (`reset_index`), never reconstruct (flat in, flat out). Safe to - adopt now: cheap, canonical, version-safe. **[provisional — internals verified]** -- **Output: linopy returns flat** — the re-stack is a cheap boundary conversion PyPSA - owns (`output` row, tested); no MI adapter lives inside linopy. -- **`n.snapshots`** — PyPSA's independent, decoupled choice: keep MI (wrap at its own - boundary) or flatten. _TBD, PyPSA-side._ +linopy internals** (`.data`, `_term`, `FILL_VALUE`) — *not* MI per se. But several are +**MI-driven workarounds** (SOC `.data.sel().roll` + `FILL_VALUE`; growth +`reindex_like(lhs.data)`) that exist because MI groupby is broken (*"internal xarray +multi-index difficulties"*, xarray#6836). #751 fixed it, so flat+aux **deletes** those +reaches; they `.sel` the *stored* MI mid-build, so they migrate regardless of how PyPSA +presents output. + +### Decision record (fill once `n.snapshots` closes) + +- **linopy drops MI from its model** — accept as input sugar, `reset_index` on entry, + never reconstruct. Safe now: cheap, canonical, version-safe. **[provisional]** +- **Output returns flat** — the re-stack is a cheap boundary conversion PyPSA owns + (`output` row, tested). +- **`n.snapshots`** — PyPSA's decoupled choice: keep MI or flatten. _TBD, PyPSA-side._ From 68f99e1e349dc8b016513103fdbc6123a0a7c6ac Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 20:01:54 +0200 Subject: [PATCH 22/33] =?UTF-8?q?docs(arithmetics):=20fix=20"MI-indexed=20?= =?UTF-8?q?results=20back"=20=E2=80=94=20linopy=20returns=20flat=20(#744)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit linopy does not hand MI-indexed results back; that contradicts flat-in/flat-out. It returns a flat solution; PyPSA re-applies its own n.snapshots index when mapping the result onto components (assign_solution already does this), so PyPSA's results stay MI-indexed -- but the re-stack is PyPSA's, not linopy's. MI is boundary sugar PyPSA owns, never inside linopy's model. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index a472078e..a1fc7472 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -9,11 +9,14 @@ ## Verdict **Feasible, and PyPSA stays stable.** Can linopy be simplified without destabilising -PyPSA, its primary consumer? Yes. PyPSA's interface does not move — it hands linopy an -MI `snapshot` `(period, timestep)` and gets MI-indexed results back; MI is **boundary -sugar** (`reset_index` in, re-stack out), never inside the model. All **seven ways -PyPSA puts that MI into linopy's model** have a flat+aux form building the *identical* -model, tested under both semantics. The change is mostly **subtraction**: +PyPSA, its primary consumer? Yes. linopy goes **flat in, flat out**: an MI `snapshot` +`(period, timestep)` is accepted only as input sugar (`reset_index`'d on entry), the +model is flat throughout, and the solution comes back flat. PyPSA re-applies its own +`n.snapshots` index when mapping that solution onto components — as `assign_solution` +already does — so PyPSA's *results* stay MI-indexed exactly as today. MI is **boundary +sugar PyPSA owns**, never inside linopy's model. All **seven ways PyPSA puts that MI +into linopy's model** have a flat+aux form building the *identical* model, tested under +both semantics. The change is mostly **subtraction**: - **less to maintain** — ~300 lines of first-class-MI machinery deleted, half one cluster (see *The payoff*). From 0f9bf75f26ff6a2857a9ea6ce7a26086b355801c Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 20:08:05 +0200 Subject: [PATCH 23/33] docs(arithmetics): resolve "no change" vs "input sugar" contradiction (#744) Separate feasibility from the entry policy. Feasibility needs no new linopy capability (the rewrites use existing ops; entry is a plain reset_index). Whether linopy accepts an MI and flattens it (input sugar, backwards-compatible) or rejects MI so callers flatten first (purest, breaks existing callers) is a boundary policy, now surfaced as an explicit Decision-record item with a recommendation (accept-as-sugar, optional deprecation path to flat-only) -- not smuggled in as both "no change" and "accept sugar" at once. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index a1fc7472..9b06e843 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -51,7 +51,10 @@ also compose into an identical LP, `test_per_period_lp_equivalent`). Glyph = | **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | ✅ | better — cheap boundary conversion (PyPSA's choice) | — | | **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | ✅ | better — removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | -No row needs a linopy change (`entry` is a user-side `reset_index`). `level groupby` +No row needs a new linopy *capability* — the rewrites use ops linopy already has. +Entry is one `reset_index`; **who** runs it — linopy (accept an MI as input sugar) or +the caller (require flat input) — is a boundary policy decided in the *Decision +record*, not a capability gap. `level groupby` is the one *necessary* row — an MI level can't be grouped (broken upstream, xarray#6836; the dense-`_term` cost is representation-agnostic, not the differentiator), flat+aux just works (#751); the rest are nicer or parity. Representation-wide, flat+aux @@ -137,8 +140,13 @@ presents output. ### Decision record (fill once `n.snapshots` closes) -- **linopy drops MI from its model** — accept as input sugar, `reset_index` on entry, - never reconstruct. Safe now: cheap, canonical, version-safe. **[provisional]** +- **linopy drops MI from its model** — flat dim + aux coords throughout, `reset_index` + on entry, never reconstruct (flat in, flat out). Safe now: cheap, canonical, + version-safe. **[provisional]** +- **MI on input — sugar or rejected?** Accept `coords=[mi]` and auto-`reset_index` + (backwards-compatible; a thin input shim survives) *vs* reject MI so callers flatten + first (purest; breaks existing MI-passing callers). _Recommend accept-as-sugar, with + an optional deprecation path to flat-only._ - **Output returns flat** — the re-stack is a cheap boundary conversion PyPSA owns (`output` row, tested). - **`n.snapshots`** — PyPSA's decoupled choice: keep MI or flatten. _TBD, PyPSA-side._ From 45f6dbe9bf40c4863587079814bd4d33e02e3e9a Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 20:13:03 +0200 Subject: [PATCH 24/33] docs(arithmetics): consolidate into a Design decisions table (#744) Replace the prose "decision record" + scattered "small additive work" mentions with one top-level Design decisions table: internal model, MI-on-input, snapshot alignment, output, n.snapshots -- each with options and a recommendation/status. Also fold in the input-policy correctness fix: MI in / flat out is a silent contract change, so silent auto-flatten is ruled out -- reject, or accept-and-warn (undecided). Slim the Verdict to point at the table; only n.snapshots is flagged as the substantive open item, and it's PyPSA's call. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 42 ++++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 9b06e843..b43cb3aa 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -9,11 +9,11 @@ ## Verdict **Feasible, and PyPSA stays stable.** Can linopy be simplified without destabilising -PyPSA, its primary consumer? Yes. linopy goes **flat in, flat out**: an MI `snapshot` -`(period, timestep)` is accepted only as input sugar (`reset_index`'d on entry), the -model is flat throughout, and the solution comes back flat. PyPSA re-applies its own -`n.snapshots` index when mapping that solution onto components — as `assign_solution` -already does — so PyPSA's *results* stay MI-indexed exactly as today. MI is **boundary +PyPSA, its primary consumer? Yes. linopy goes **flat in, flat out**: the model is flat +throughout and the solution comes back flat. An MI `snapshot` `(period, timestep)` is +flattened on entry (`reset_index`); PyPSA re-applies its own `n.snapshots` index when +mapping that solution onto components — as `assign_solution` already does — so PyPSA's +*results* stay MI-indexed exactly as today. MI is **boundary sugar PyPSA owns**, never inside linopy's model. All **seven ways PyPSA puts that MI into linopy's model** have a flat+aux form building the *identical* model, tested under both semantics. The change is mostly **subtraction**: @@ -25,9 +25,10 @@ both semantics. The change is mostly **subtraction**: - **unblocks work** — the MI-level groupby broken upstream (xarray#6836) works flat (#751); the MI coupling forcing PyPSA into linopy internals ([#752](https://github.com/PyPSA/linopy/issues/752)) goes. -Small additive work: positional snapshot alignment; shrink MI-input to -accept-then-`reset_index`. **One open item, PyPSA's own decoupled call:** whether -`n.snapshots` stays an MI (a cheap boundary wrap) — linopy works either way. +The choices this forces are tabled in *Design decisions* — most settled pending +adoption. The substantive open one, and the only one not linopy's, is PyPSA's +**`n.snapshots`**: keep the MI (a cheap boundary wrap) or flatten — linopy works either +way. *Proof set, not universal: PyPSA is the one consumer audited. But after `reset_index` (general to any MI) everything is ordinary xarray, so the only MI-specific capability @@ -95,6 +96,18 @@ levels. > _Strip from a read-only sweep of the v1 tree; counts order-of-magnitude, not yet > executed._ +## Design decisions + +What adopting flat+aux actually decides — **recommendation** in the last column. + +| decision | options | recommendation | +|---|---|---| +| **Internal model** | first-class MI · **flat dim + aux coords** | flat+aux — feasible, tested, mostly subtraction. **[provisional]** | +| **MI on input** | reject · accept-with-warning · ~~silent accept~~ | silent flatten is **ruled out** (MI-in/flat-out surprises): reject, or accept *and warn*. _Undecided._ | +| **Snapshot alignment** | tuple-identity · **positional** | positional — one canonical `n.snapshots` order, matching a plain datetime snapshot (§11). | +| **Output** | reconstruct MI · **return flat** | flat — re-stack is the caller's cheap boundary step (`output` row, tested). | +| **`n.snapshots`** *(PyPSA-side)* | keep MI · flatten | PyPSA's decoupled call; linopy works either way. _TBD._ | + ## Appendix ### `reset_index` is the whole transform @@ -137,16 +150,3 @@ linopy internals** (`.data`, `_term`, `FILL_VALUE`) — *not* MI per se. But sev multi-index difficulties"*, xarray#6836). #751 fixed it, so flat+aux **deletes** those reaches; they `.sel` the *stored* MI mid-build, so they migrate regardless of how PyPSA presents output. - -### Decision record (fill once `n.snapshots` closes) - -- **linopy drops MI from its model** — flat dim + aux coords throughout, `reset_index` - on entry, never reconstruct (flat in, flat out). Safe now: cheap, canonical, - version-safe. **[provisional]** -- **MI on input — sugar or rejected?** Accept `coords=[mi]` and auto-`reset_index` - (backwards-compatible; a thin input shim survives) *vs* reject MI so callers flatten - first (purest; breaks existing MI-passing callers). _Recommend accept-as-sugar, with - an optional deprecation path to flat-only._ -- **Output returns flat** — the re-stack is a cheap boundary conversion PyPSA owns - (`output` row, tested). -- **`n.snapshots`** — PyPSA's decoupled choice: keep MI or flatten. _TBD, PyPSA-side._ From 2ff3fc6984578e8ad3e19ed69ecac5d7ea6515ac Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 20:17:16 +0200 Subject: [PATCH 25/33] =?UTF-8?q?docs(arithmetics):=20MI=20surface=20is=20?= =?UTF-8?q?bigger=20=E2=80=94=20two-layer=20payoff=20+=20corrected=20sweep?= =?UTF-8?q?=20(#744)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ~300 estimate was a floor. Two corrections from a deeper measure: - assign_multiindex_safe (the #303 "would corrupt the index" workaround, 39 call sites) is snapshot-MI-driven, not a general helper: internal stacks use create_index=False, so the snapshot MI is the only thing that triggers the corruption warning. Under flat+aux all 39 sites revert to plain .assign(). The first sweep wrongly marked it "stays" -- corrected, with a strip-table row. - Add the diffuse "cognitive tax" layer the line-count misses: 6 isinstance(MI) guards, 17 level-ops, ~40 quirk-comments, and 169 lines of MI edge-case tests across 10 files. Reframe the mental-model pillar around that defensive surface (guards, the 39-site corruption workaround, quirk-comments, edge-case tests) rather than just lines. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 36 +++++++++++++------- 1 file changed, 23 insertions(+), 13 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index b43cb3aa..6c7d439e 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -21,7 +21,9 @@ both semantics. The change is mostly **subtraction**: - **less to maintain** — ~300 lines of first-class-MI machinery deleted, half one cluster (see *The payoff*). - **xarray-native internals** — linopy stops knowing `pd.MultiIndex` exists or covering - its quirks; the model is just dims + ordinary aux coords. + its quirks: the `isinstance(MI)` guards, the **39-site** *"would corrupt the index"* + workaround (#303), the ~40 quirk-comments, the 169 lines of MI edge-case tests — a + whole defensive surface gone. The model is just dims + ordinary aux coords. - **unblocks work** — the MI-level groupby broken upstream (xarray#6836) works flat (#751); the MI coupling forcing PyPSA into linopy internals ([#752](https://github.com/PyPSA/linopy/issues/752)) goes. @@ -78,20 +80,28 @@ Pinned by `test_variable_mi_tuple_sel_not_forwarded`.) ## The payoff -Adopting flat+aux **deletes** ~300 lines of first-class-MI machinery — concentrated, -not diffuse: ~half is the `alignment.py` *level-projection* subsystem, whose only job -is to make a single MI level align like a dimension (dead once levels are aux coords). +Adopting flat+aux deletes first-class-MI machinery on **two layers**. The +*concentrated* one is ~300 lines — ~half the `alignment.py` *level-projection* +subsystem, whose only job is to make a single MI level align like a dimension (dead +once levels are aux coords): -| strip | what it does today | ~lines | +| strip | what it does today | scale | |---|---|---| -| **`alignment.py` level-projection** — `_project_onto_multiindex_levels`, `_enforce_implicit_projections`, `_LevelProjection`, the `projections` plumbing through `broadcast_to_coords`, plus the MI branches in `_expand_missing_dims`/`validate_alignment` and `_as_multiindex` | align a single-level operand against a full MI dim | ~150 | -| **netcdf MI (de)serialization** — `io.py` flatten-on-write + reconstruct (`{dim}_multiindex` attr); `common.py` MI level/code (de)serialize | flatten MI to store, rebuild MI on read | ~50 (keep a read-only shim for old `.nc`) | -| **scattered MI guards/branches** in alignment & coords | skip-logic that only fires when a dim is an MI | remainder | - -Stays: `assign_multiindex_safe` and internal `_term`/`_factor`/groupby stacking -(general helpers over linopy's own indexes, unrelated to snapshot-MI), and the §11 -aux-conflict logic — which gets *more* central, aux coords being the new home for the -levels. +| **`alignment.py` level-projection** — `_project_onto_multiindex_levels`, `_enforce_implicit_projections`, `_LevelProjection`, the `projections` plumbing through `broadcast_to_coords`, plus the MI branches in `_expand_missing_dims`/`validate_alignment` and `_as_multiindex` | align a single-level operand against a full MI dim | ~150 lines | +| **netcdf MI (de)serialization** — `io.py` flatten-on-write + reconstruct (`{dim}_multiindex` attr); `common.py` MI level/code (de)serialize | flatten MI to store, rebuild MI on read | ~50 lines (+ read-only shim for old `.nc`) | +| **`assign_multiindex_safe`** (#303) — the corruption workaround | rebuild Datasets to dodge the *"would corrupt the index"* warning on assign — which fires *only* for the snapshot MI (internal stacks are `create_index=False`, so no other MI exists) | **39 call sites** → plain `.assign()` | +| **scattered guards / level-ops** in alignment & coords | skip-logic that only fires when a dim is an MI | 6 `isinstance(MI)` + 17 | + +The *diffuse* layer is the **cognitive tax** that doesn't show up as deletable lines: +~40 quirk-comments explaining MI *"difficulties"*, and **169 lines of MI edge-case +tests across 10 files** — all moot once a snapshot is a flat dim. (This is why the +first sweep undercounted: it scored deletable clusters and conservatively kept +`assign_multiindex_safe` — but with the snapshot MI gone it is the sole consumer of +the #303 workaround, so it goes too.) + +Stays: the internal `_term`/`_factor`/groupby stacking (`create_index=False`, not MIs) +and the §11 aux-conflict logic — which gets *more* central, aux coords being the new +home for the levels. > _Strip from a read-only sweep of the v1 tree; counts order-of-magnitude, not yet > executed._ From ad1f5c7229d7fb6e05f30dddb841f4a9e283027e Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 20:19:18 +0200 Subject: [PATCH 26/33] docs(arithmetics): add latent-risk and MI-not-promoted arguments (#744) Two more points strengthening the case: - Latent risk: MI's edge surface almost certainly isn't fully covered (the #303 corruption is one gap that already bit), so keeping MI carries unknown-bug risk that flat+aux retires. Folded into the mental-model pillar and the payoff. - MI isn't a promoted feature, so few users lean on it -- lowers the cost of the lost level-tuple .sel and makes the "no other infeasible case" generalization safer. Folded into the proof-set caveat. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 6c7d439e..2cdcb2ad 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -23,7 +23,8 @@ both semantics. The change is mostly **subtraction**: - **xarray-native internals** — linopy stops knowing `pd.MultiIndex` exists or covering its quirks: the `isinstance(MI)` guards, the **39-site** *"would corrupt the index"* workaround (#303), the ~40 quirk-comments, the 169 lines of MI edge-case tests — a - whole defensive surface gone. The model is just dims + ordinary aux coords. + whole defensive surface gone, along with the latent risk of the MI edge cases it + never fully covered. The model is just dims + ordinary aux coords. - **unblocks work** — the MI-level groupby broken upstream (xarray#6836) works flat (#751); the MI coupling forcing PyPSA into linopy internals ([#752](https://github.com/PyPSA/linopy/issues/752)) goes. @@ -34,7 +35,8 @@ way. *Proof set, not universal: PyPSA is the one consumer audited. But after `reset_index` (general to any MI) everything is ordinary xarray, so the only MI-specific capability -lost for any user is `.sel` by level tuple (→ `where`). Counterexamples welcome.* +lost for any user is `.sel` by level tuple (→ `where`) — and MI snapshots are not a +feature linopy promotes, so few lean on them. Counterexamples welcome.* ## The evidence @@ -94,8 +96,10 @@ once levels are aux coords): The *diffuse* layer is the **cognitive tax** that doesn't show up as deletable lines: ~40 quirk-comments explaining MI *"difficulties"*, and **169 lines of MI edge-case -tests across 10 files** — all moot once a snapshot is a flat dim. (This is why the -first sweep undercounted: it scored deletable clusters and conservatively kept +tests across 10 files** — all moot once a snapshot is a flat dim. And that surface +almost certainly doesn't cover MI's *full* edge behaviour (the #303 corruption is one +gap that already bit), so every uncovered MI corner is **latent risk** flat+aux simply +retires. (This is why the first sweep undercounted: it scored deletable clusters and conservatively kept `assign_multiindex_safe` — but with the snapshot MI gone it is the sole consumer of the #303 workaround, so it goes too.) From 65fe02330ebdcd199a15871eed5569ce233f4bb8 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 20:31:00 +0200 Subject: [PATCH 27/33] =?UTF-8?q?docs(arithmetics):=20add=20the=20second?= =?UTF-8?q?=20MI=20surface=20=E2=80=94=20multi-key=20groupby=20(#744)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "MI in exactly one place — snapshot" claim was false. Verified here (both semantics): a multi-key groupby (DataFrame grouper) mints a stacked `group` MultiIndex, and PyPSA-Eur's add_CCL_constraints consumes it richly (groupby(country×carrier).sum() then .intersection / .loc[MI] / .sel(group=index), solve_network.py L1064/L1080-1095). add_EQ/add_BAU are single-key (no group MI) -- correcting an over-broad relayed claim. Integrate at "2nd open item, flagged harder": scope the Verdict's snapshot claim, turn one open item into two (n.snapshots + multi-key groupby), add a "Second MI surface" section and a Design-decisions row, and make the proof-set caveat honest (the generalization already cracked once). Framing per FBumann: it's the same flat+aux change, linopy-owned (groupby returns flat group + key aux coords) -- but a public-API change with external consumers, so it needs a deprecation path, not a boundary fix. Blast radius (-Earth/-DE/plugins) not yet scoped. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 59 +++++++++++++++----- 1 file changed, 46 insertions(+), 13 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 2cdcb2ad..3b6bda4f 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -13,10 +13,10 @@ PyPSA, its primary consumer? Yes. linopy goes **flat in, flat out**: the model i throughout and the solution comes back flat. An MI `snapshot` `(period, timestep)` is flattened on entry (`reset_index`); PyPSA re-applies its own `n.snapshots` index when mapping that solution onto components — as `assign_solution` already does — so PyPSA's -*results* stay MI-indexed exactly as today. MI is **boundary -sugar PyPSA owns**, never inside linopy's model. All **seven ways PyPSA puts that MI -into linopy's model** have a flat+aux form building the *identical* model, tested under -both semantics. The change is mostly **subtraction**: +*results* stay MI-indexed exactly as today. For the **snapshot** axis MI is boundary +sugar PyPSA owns, never inside linopy's model; all **seven ways PyPSA puts it into +linopy's model** have a flat+aux form building the *identical* model, tested under both +semantics. The change is mostly **subtraction**: - **less to maintain** — ~300 lines of first-class-MI machinery deleted, half one cluster (see *The payoff*). @@ -29,19 +29,27 @@ both semantics. The change is mostly **subtraction**: (#751); the MI coupling forcing PyPSA into linopy internals ([#752](https://github.com/PyPSA/linopy/issues/752)) goes. The choices this forces are tabled in *Design decisions* — most settled pending -adoption. The substantive open one, and the only one not linopy's, is PyPSA's -**`n.snapshots`**: keep the MI (a cheap boundary wrap) or flatten — linopy works either -way. - -*Proof set, not universal: PyPSA is the one consumer audited. But after `reset_index` -(general to any MI) everything is ordinary xarray, so the only MI-specific capability -lost for any user is `.sel` by level tuple (→ `where`) — and MI snapshots are not a -feature linopy promotes, so few lean on them. Counterexamples welcome.* +adoption. **Two are genuinely open, the second the harder:** + +1. **`n.snapshots`** — PyPSA's own decoupled call: keep the MI (a cheap boundary wrap) + or flatten; linopy works either way. +2. **multi-key `groupby`** — the *one* place linopy itself mints an MI (not PyPSA): a + multi-key grouper returns a stacked `group` MultiIndex, consumed *downstream* (see + *Second MI surface*). The same flat+aux fix, linopy-owned — but on a **public API** + with **external** consumers, so it needs a deprecation path, not a boundary fix. + +*Proof set, not universal: a PyPSA-Eur spot-check already surfaced a real second case +(multi-key `groupby`, below), so the generalization is not free — other forks/plugins +are unswept. For the **snapshot** axis specifically, after `reset_index` everything is +ordinary xarray and the only MI-specific capability lost is `.sel` by level tuple (→ +`where`); MI snapshots aren't promoted, so few lean on them. More counterexamples +welcome — audit your own multi-key `groupby` + `.sel(group=…)`.* ## The evidence PyPSA's observed MI uses of linopy, split by whether the MI **enters the model**. -In-model uses are the matrix — all ✅ (tested under `legacy`+`v1`; build-time rewrites +The **snapshot** surface is the matrix below (the second surface, multi-key `groupby`, +has its own section). All ✅ (tested under `legacy`+`v1`; build-time rewrites also compose into an identical LP, `test_per_period_lp_equivalent`). Glyph = **feasible** (✅ tested · 🔲 achievable · ❌ no); word = **desirable** (better · parity · worse). @@ -80,6 +88,30 @@ but not inside linopy. [#752](https://github.com/PyPSA/linopy/issues/752) §2; flat+aux makes it `where`/`isel`. Pinned by `test_variable_mi_tuple_sel_not_forwarded`.) +## Second MI surface: multi-key `groupby` + +The matrix above is the snapshot MI — passed *in* by PyPSA, solved at the entry +boundary. There is a **second** in-linopy MI, and it is **linopy's own**: a multi-key +`groupby` mints a stacked `group` MultiIndex. Verified here under both semantics — +`p.groupby().sum()` → the `group` dim is a `pd.MultiIndex`, +and `.sel(group=("DE","wind"))` works. + +PyPSA-Eur's `add_CCL_constraints` consumes it richly (not just `.sel`): +`grouper = concat([country, carrier]); lhs = p_nom.groupby(grouper).sum()` +([`solve_network.py` L1064](https://github.com/PyPSA/pypsa-eur/blob/master/scripts/solve_network.py)), +then `minimum.indexes["group"].intersection(lhs.indexes["group"])` and +`lhs.sel(group=index) >= minimum.loc[index]` (L1080‑1095). (`add_EQ`/`add_BAU` group by +a *single* key — flat `group` dim, no MI.) + +It is the **same flat+aux change, owned by linopy**: `groupby` returns a flat `group` +dim with `country`/`carrier` as aux coords — uniform with every other dim, deleting the +last place linopy mints an MI (✅ desirable: safer / flexible / one representation; ⚪ +parity on capability). But unlike snapshot it is a **public-API** change with +**external** consumers (PyPSA-Eur/-Earth/-DE), so the downstream `.sel(group=tuple)` / +`.intersection` / `.loc[MI]` → aux-coord selection migration needs a **deprecation +path**, and the full blast radius (forks, plugins) is **not yet scoped**. The harder of +the two open items. + ## The payoff Adopting flat+aux deletes first-class-MI machinery on **two layers**. The @@ -121,6 +153,7 @@ What adopting flat+aux actually decides — **recommendation** in the last colum | **Snapshot alignment** | tuple-identity · **positional** | positional — one canonical `n.snapshots` order, matching a plain datetime snapshot (§11). | | **Output** | reconstruct MI · **return flat** | flat — re-stack is the caller's cheap boundary step (`output` row, tested). | | **`n.snapshots`** *(PyPSA-side)* | keep MI · flatten | PyPSA's decoupled call; linopy works either way. _TBD._ | +| **multi-key `groupby` output** | stacked `group` MI · **flat `group` + key aux coords** | flat+aux (uniform, deletes the last MI linopy mints) — but a **public-API** change; needs a deprecation path for external `.sel(group=…)` consumers. _Open; blast radius unscoped._ | ## Appendix From 42054af9515998af4d6e6377ddc062a3a19f6c27 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 20:36:23 +0200 Subject: [PATCH 28/33] =?UTF-8?q?docs(arithmetics):=20fold=20multi-key=20g?= =?UTF-8?q?roupby=20into=20the=20matrix=20as=20the=20=F0=9F=94=B2=20row=20?= =?UTF-8?q?(#744)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per FBumann: now that the deprecation is just the standard legacy/v1 path (warn under legacy, raise/flat+aux under v1), multi-key groupby is the same kind of in-linopy MI as the snapshot rows -- siloing it overstated its difference. Move it into table 1 as the 8th row, marked 🔲 (the one achievable-not-yet-tested row, vs the 7 tested ✅ -- which also finally exercises the glyph legend). Replace the standalone "Second MI surface" section with a tight caption (verified both semantics; PyPSA-Eur CCL cite; legacy/v1 migration; blast radius the only open part). Soften the earlier "harder / needs a deprecation path" framing throughout: it reaches external code but rides the standard switch, so it's guided, not a hard break. Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/multiindex-feasibility.md | 59 ++++++++------------ 1 file changed, 24 insertions(+), 35 deletions(-) diff --git a/arithmetics-design/multiindex-feasibility.md b/arithmetics-design/multiindex-feasibility.md index 3b6bda4f..743aff21 100644 --- a/arithmetics-design/multiindex-feasibility.md +++ b/arithmetics-design/multiindex-feasibility.md @@ -29,14 +29,16 @@ semantics. The change is mostly **subtraction**: (#751); the MI coupling forcing PyPSA into linopy internals ([#752](https://github.com/PyPSA/linopy/issues/752)) goes. The choices this forces are tabled in *Design decisions* — most settled pending -adoption. **Two are genuinely open, the second the harder:** +adoption. **Two are genuinely open** (the second reaches external code, but via the +standard `legacy`/`v1` deprecation): 1. **`n.snapshots`** — PyPSA's own decoupled call: keep the MI (a cheap boundary wrap) or flatten; linopy works either way. 2. **multi-key `groupby`** — the *one* place linopy itself mints an MI (not PyPSA): a - multi-key grouper returns a stacked `group` MultiIndex, consumed *downstream* (see - *Second MI surface*). The same flat+aux fix, linopy-owned — but on a **public API** - with **external** consumers, so it needs a deprecation path, not a boundary fix. + multi-key grouper returns a stacked `group` MultiIndex, consumed *downstream* (the + **multi-key `groupby`** matrix row). Same flat+aux fix, linopy-owned; external `.sel(group=…)` + consumers migrate via the `legacy`/`v1` switch (warn → raise) — a guided migration, + not a hard break. Open part: the blast radius. *Proof set, not universal: a PyPSA-Eur spot-check already surfaced a real second case (multi-key `groupby`, below), so the generalization is not free — other forks/plugins @@ -47,12 +49,12 @@ welcome — audit your own multi-key `groupby` + `.sel(group=…)`.* ## The evidence -PyPSA's observed MI uses of linopy, split by whether the MI **enters the model**. -The **snapshot** surface is the matrix below (the second surface, multi-key `groupby`, -has its own section). All ✅ (tested under `legacy`+`v1`; build-time rewrites -also compose into an identical LP, `test_per_period_lp_equivalent`). Glyph = -**feasible** (✅ tested · 🔲 achievable · ❌ no); word = **desirable** (better · parity -· worse). +linopy's **in-model** MI uses. The first seven are the **snapshot** surface (PyPSA +passes the MI *in*) — all ✅, tested under `legacy`+`v1` and shown to compose into an +identical LP (`test_per_period_lp_equivalent`). The eighth is **multi-key `groupby`** — +the one MI *linopy itself mints* — 🔲 achievable (a linopy change + a downstream +migration, captioned below). Glyph = **feasible** (✅ tested · 🔲 achievable · ❌ no); +word = **desirable** (better · parity · worse). | op | MI form | flat+aux form | feasible | desirable | PyPSA call site @ v1.2.4 | |---|---|---|---|---|---| @@ -63,6 +65,7 @@ also compose into an identical LP, `test_per_period_lp_equivalent`). Glyph = | **storage SOC** | `.data.sel().roll` + `FILL_VALUE` rebuild; `_period_start_mask` (shared w/ ramps) | previous-SOC via `groupby("period").roll`, then period-start: wrap (cyclic) · `.where` term (non-cyclic) · `mask=` row (ramp) | ✅ | better — deletes `FILL_VALUE` hack | [roll L1694](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1694), [fill L1735‑1737](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1735-L1737), [store-energy L1875‑1908](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L1875-L1908); boundary mask [`common.py` L22](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/common.py#L22) → also ramps [`constraints.py` L838](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/constraints.py#L838) | | **output** | `solution`/`dual` MI-indexed | flat solution; caller re-stacks (or not) | ✅ | better — cheap boundary conversion (PyPSA's choice) | — | | **snapshots param** | MI parked on `model.parameters`, rebuilt via `.to_index()` | flat param; `assign_solution` rebuilds `period`/`timestep` from aux | ✅ | better — removes the MI living *inside* a linopy object | store [`optimize.py` L689](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L689); rebuild [L905](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L905)/[L1114](https://github.com/PyPSA/PyPSA/blob/v1.2.4/pypsa/optimization/optimize.py#L1114) | +| **multi-key `groupby`** *(linopy-minted)* | `groupby(country×carrier).sum()` → stacked `group` MI; consumed `.sel(group=…)`/`.intersection`/`.loc` | `groupby` → flat `group` + `country`/`carrier` aux coords; select on aux | 🔲 | better — uniform, deletes the last MI linopy mints; **public API**, consumers migrate via `legacy`→`v1` (warn→raise) | pypsa-eur [`solve_network.py` L1064/L1080‑1095](https://github.com/PyPSA/pypsa-eur/blob/master/scripts/solve_network.py) (CCL) | No row needs a new linopy *capability* — the rewrites use ops linopy already has. Entry is one `reset_index`; **who** runs it — linopy (accept an MI as input sugar) or @@ -75,6 +78,16 @@ is also **safer** (v1 *raises* on a conflicting aux coord via `enforce_aux_confl MI only hides it, [#295](https://github.com/PyPSA/linopy/issues/295)) and **flexible** (level coords `drop_vars`/`rename` freely; an MI raises *"would corrupt the index"*). +The **multi-key `groupby`** row is the one MI *linopy itself* mints and the one an +*external* model consumes: PyPSA-Eur's `add_CCL_constraints` does +`groupby(country×carrier).sum()` then `.intersection`/`.loc[MI]`/`.sel(group=index)` +([`solve_network.py` L1064/L1080‑1095](https://github.com/PyPSA/pypsa-eur/blob/master/scripts/solve_network.py); +`add_EQ`/`add_BAU` are single-key, no MI). Verified here both semantics. The fix is +linopy-owned and *the same flat+aux change* — `groupby` returns a flat `group` dim + +key aux coords — and rides the `legacy`/`v1` switch (mint+warn under `legacy`, +flat+aux+raise under `v1`), so consumers move on their own schedule. The only open part +is the **blast radius** (forks, plugins), unswept. + **Boundary uses** — two MI usages need no in-linopy solution, failing the in-model test on opposite axes: `stochastic` is inside linopy but not an MI; `n.snapshots` is an MI but not inside linopy. @@ -88,30 +101,6 @@ but not inside linopy. [#752](https://github.com/PyPSA/linopy/issues/752) §2; flat+aux makes it `where`/`isel`. Pinned by `test_variable_mi_tuple_sel_not_forwarded`.) -## Second MI surface: multi-key `groupby` - -The matrix above is the snapshot MI — passed *in* by PyPSA, solved at the entry -boundary. There is a **second** in-linopy MI, and it is **linopy's own**: a multi-key -`groupby` mints a stacked `group` MultiIndex. Verified here under both semantics — -`p.groupby().sum()` → the `group` dim is a `pd.MultiIndex`, -and `.sel(group=("DE","wind"))` works. - -PyPSA-Eur's `add_CCL_constraints` consumes it richly (not just `.sel`): -`grouper = concat([country, carrier]); lhs = p_nom.groupby(grouper).sum()` -([`solve_network.py` L1064](https://github.com/PyPSA/pypsa-eur/blob/master/scripts/solve_network.py)), -then `minimum.indexes["group"].intersection(lhs.indexes["group"])` and -`lhs.sel(group=index) >= minimum.loc[index]` (L1080‑1095). (`add_EQ`/`add_BAU` group by -a *single* key — flat `group` dim, no MI.) - -It is the **same flat+aux change, owned by linopy**: `groupby` returns a flat `group` -dim with `country`/`carrier` as aux coords — uniform with every other dim, deleting the -last place linopy mints an MI (✅ desirable: safer / flexible / one representation; ⚪ -parity on capability). But unlike snapshot it is a **public-API** change with -**external** consumers (PyPSA-Eur/-Earth/-DE), so the downstream `.sel(group=tuple)` / -`.intersection` / `.loc[MI]` → aux-coord selection migration needs a **deprecation -path**, and the full blast radius (forks, plugins) is **not yet scoped**. The harder of -the two open items. - ## The payoff Adopting flat+aux deletes first-class-MI machinery on **two layers**. The @@ -153,7 +142,7 @@ What adopting flat+aux actually decides — **recommendation** in the last colum | **Snapshot alignment** | tuple-identity · **positional** | positional — one canonical `n.snapshots` order, matching a plain datetime snapshot (§11). | | **Output** | reconstruct MI · **return flat** | flat — re-stack is the caller's cheap boundary step (`output` row, tested). | | **`n.snapshots`** *(PyPSA-side)* | keep MI · flatten | PyPSA's decoupled call; linopy works either way. _TBD._ | -| **multi-key `groupby` output** | stacked `group` MI · **flat `group` + key aux coords** | flat+aux (uniform, deletes the last MI linopy mints) — but a **public-API** change; needs a deprecation path for external `.sel(group=…)` consumers. _Open; blast radius unscoped._ | +| **multi-key `groupby` output** | stacked `group` MI · **flat `group` + key aux coords** | flat+aux (uniform, deletes the last MI linopy mints); external `.sel(group=…)` consumers migrate via the `legacy`/`v1` switch (warn → raise). _Open part: blast radius._ | ## Appendix From 1c167d49c9cf8533c4cb123daa6f2143564aa143 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 21:37:39 +0200 Subject: [PATCH 29/33] feat(v1): multi-key groupby returns a flat group dim under v1 (#744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A frame grouper (`groupby(DataFrame).sum()`, e.g. PyPSA-Eur's CCL country×carrier) mints a stacked `group` MultiIndex today. Under v1 it now returns a flat `group` dim with the keys as ordinary aux coords — uniform with every other dim, no MI; select via groupby/where, not a `.sel(group=tuple)` level tuple. Legacy keeps the stacked MultiIndex and emits a DeprecationWarning pointing at the change. Scope: only the cases where a stacked group MI survives today — the DataFrame grouper and `observed=True`. The default `observed=False` list-of-names path that already unstacks into separate dims is untouched. The legacy deprecation warning is limited to the user-facing frame grouper. This is the first transition slice for dropping first-class MultiIndex in v1 (the strip of the now-dead-under-v1 MI machinery follows at legacy EOL). Affected groupby tests split into @pytest.mark.legacy (MI + asserts the warning) / @pytest.mark.v1 (flat+aux) twins. Full suite green under both semantics. Co-Authored-By: Claude Opus 4.8 (1M context) --- linopy/expressions.py | 16 +++- linopy/semantics.py | 15 +++ test/test_linear_expression.py | 161 +++++++++++++++++++++------------ 3 files changed, 131 insertions(+), 61 deletions(-) diff --git a/linopy/expressions.py b/linopy/expressions.py index dd0a72d1..5d352658 100644 --- a/linopy/expressions.py +++ b/linopy/expressions.py @@ -91,6 +91,7 @@ from linopy.semantics import ( _legacy_coord_mismatch_message, _legacy_coord_reorder_message, + _legacy_group_multiindex_message, _legacy_nan_rhs_constraint_message, _shared_dim_mismatch_message, absorb_absence, @@ -378,10 +379,19 @@ def sum( if int_map is not None: index = ds.indexes[GROUP_DIM].map({v: k for k, v in int_map.items()}) - index.names = [str(col) for col in orig_group.columns] + level_names = [str(col) for col in orig_group.columns] + index.names = level_names index.name = GROUP_DIM - new_coords = Coordinates.from_pandas_multiindex(index, GROUP_DIM) - ds = ds.assign_coords(new_coords) + stacked_survives = multikey_frame is None or observed + if stacked_survives and is_v1(): # flat group dim + keys as aux coords + ds = ds.assign_coords( + {n: (GROUP_DIM, index.get_level_values(n)) for n in level_names} + ) + else: + if multikey_frame is None: # user-facing frame grouper + warn_legacy(_legacy_group_multiindex_message(level_names)) + new_coords = Coordinates.from_pandas_multiindex(index, GROUP_DIM) + ds = ds.assign_coords(new_coords) ds = ds.rename({GROUP_DIM: final_group_name}) if multikey_frame is not None and not observed: diff --git a/linopy/semantics.py b/linopy/semantics.py index 38e06695..88f30b8f 100644 --- a/linopy/semantics.py +++ b/linopy/semantics.py @@ -217,6 +217,21 @@ def _legacy_masked_variable_message(name: str) -> str: ) +def _legacy_group_multiindex_message(level_names: list[str]) -> str: + """A multi-key groupby returns a stacked ``group`` MultiIndex under legacy.""" + levels = ", ".join(repr(n) for n in level_names) + return ( + f"A groupby with a frame grouper returned a stacked `group` MultiIndex over" + f" ({levels})." + " Under legacy the `group` dim is that MultiIndex, so `.sel(group=(...))` by" + " tuple works. Under v1 the `group` dim is flat with those keys as ordinary" + f" aux coords instead — select via `.groupby`/`.where` on {levels}, not a" + " level tuple." + f"\n Resolve: replace `.sel(group=(...))` with selection on the" + f" {levels} aux coord(s)." + _OPT_IN_HINT + ) + + _LINOPY_ROOT = os.path.dirname(os.path.abspath(__file__)) diff --git a/test/test_linear_expression.py b/test/test_linear_expression.py index 1f135e86..1005152a 100644 --- a/test/test_linear_expression.py +++ b/test/test_linear_expression.py @@ -1459,17 +1459,26 @@ def test_sparse_combination_filled(self) -> None: assert (cell.vars == -1).all() assert cell.coeffs.isnull().all() + @pytest.mark.legacy def test_dataframe_grouper_stays_compact(self) -> None: - # the DataFrame grouper keeps the stacked observed-only group dim + # legacy: the DataFrame grouper keeps the stacked observed-only group MI expr = self._expr([2020, 2020, 2030, 2030], list("wwws")) df = expr.data[["period", "season"]].to_dataframe()[["period", "season"]] - - grouped = expr.groupby(df).sum() - - assert "group" in grouped.dims + with pytest.warns(LinopySemanticsWarning, match=r"stacked `group` MultiIndex"): + grouped = expr.groupby(df).sum() assert isinstance(grouped.data.indexes["group"], pd.MultiIndex) assert grouped.sizes["group"] == 3 # observed, not the 2x2=4 grid + @pytest.mark.v1 + def test_dataframe_grouper_stays_compact_v1(self) -> None: + # v1: flat `group` dim + period/season aux coords, still observed-only + expr = self._expr([2020, 2020, 2030, 2030], list("wwws")) + df = expr.data[["period", "season"]].to_dataframe()[["period", "season"]] + grouped = expr.groupby(df).sum() + assert not isinstance(grouped.data.indexes["group"], pd.MultiIndex) + assert {"period", "season"} <= set(grouped.data.coords) + assert grouped.sizes["group"] == 3 + def test_blowup_warns_when_sparse(self) -> None: # 200 observed combos, 200x200 grid -> nudge toward observed=True expr = self._expr(list(range(200)), list(range(200))) @@ -1767,89 +1776,125 @@ def test_linear_expression_groupby_with_series_on_multiindex( assert grouped.nterm == len_grouped_dim -@pytest.mark.parametrize("use_fallback", [True, False]) -def test_linear_expression_groupby_with_dataframe( - v: Variable, use_fallback: bool -) -> None: +@pytest.mark.legacy +def test_linear_expression_groupby_with_dataframe(v: Variable) -> None: expr = 1 * v groups = pd.DataFrame( {"a": [1] * 10 + [2] * 10, "b": list(range(4)) * 5}, index=v.indexes["dim_2"] ) - if use_fallback: - with pytest.raises(ValueError): - expr.groupby(groups).sum(use_fallback=use_fallback) - return + with pytest.raises(ValueError): + expr.groupby(groups).sum(use_fallback=True) + with pytest.warns(LinopySemanticsWarning, match=r"stacked `group` MultiIndex"): + grouped = expr.groupby(groups).sum() + assert isinstance(grouped.indexes["group"], pd.MultiIndex) + assert set(grouped.data.group.values) == set( + pd.MultiIndex.from_frame(groups).values + ) + assert grouped.nterm == 3 - grouped = expr.groupby(groups).sum(use_fallback=use_fallback) - index = pd.MultiIndex.from_frame(groups) - assert "group" in grouped.dims - assert set(grouped.data.group.values) == set(index.values) + +@pytest.mark.v1 +def test_linear_expression_groupby_with_dataframe_v1(v: Variable) -> None: + """v1: a frame grouper yields a flat `group` dim with the keys as aux coords.""" + expr = 1 * v + groups = pd.DataFrame( + {"a": [1] * 10 + [2] * 10, "b": list(range(4)) * 5}, index=v.indexes["dim_2"] + ) + with pytest.raises(ValueError): + expr.groupby(groups).sum(use_fallback=True) + grouped = expr.groupby(groups).sum() + assert not isinstance(grouped.indexes["group"], pd.MultiIndex) + keys = set(zip(grouped.data.a.values, grouped.data.b.values)) + assert keys == set(pd.MultiIndex.from_frame(groups).values) assert grouped.nterm == 3 -@pytest.mark.parametrize("use_fallback", [True, False]) +@pytest.mark.legacy def test_linear_expression_groupby_with_dataframe_with_same_group_name( - v: Variable, use_fallback: bool + v: Variable, ) -> None: - """ - Test that the group by works with a dataframe whose column name is the same as - the dimension to group. - """ + """A frame grouper whose column name equals the grouped dimension.""" expr = 1 * v groups = pd.DataFrame( {"dim_2": [1] * 10 + [2] * 10, "b": list(range(4)) * 5}, index=v.indexes["dim_2"], ) - if use_fallback: - with pytest.raises(ValueError): - expr.groupby(groups).sum(use_fallback=use_fallback) - return - - grouped = expr.groupby(groups).sum(use_fallback=use_fallback) - index = pd.MultiIndex.from_frame(groups) - assert "group" in grouped.dims - assert set(grouped.data.group.values) == set(index.values) + with pytest.raises(ValueError): + expr.groupby(groups).sum(use_fallback=True) + with pytest.warns(LinopySemanticsWarning, match=r"stacked `group` MultiIndex"): + grouped = expr.groupby(groups).sum() + assert set(grouped.data.group.values) == set( + pd.MultiIndex.from_frame(groups).values + ) assert grouped.nterm == 3 -@pytest.mark.parametrize("use_fallback", [True, False]) -def test_linear_expression_groupby_with_dataframe_on_multiindex( - u: Variable, use_fallback: bool +@pytest.mark.v1 +def test_linear_expression_groupby_with_dataframe_with_same_group_name_v1( + v: Variable, ) -> None: - expr = 1 * u - len_grouped_dim = len(u.data["dim_3"]) - groups = pd.DataFrame({"a": [1] * len_grouped_dim}, index=u.indexes["dim_3"]) + expr = 1 * v + groups = pd.DataFrame( + {"dim_2": [1] * 10 + [2] * 10, "b": list(range(4)) * 5}, + index=v.indexes["dim_2"], + ) + grouped = expr.groupby(groups).sum() + assert not isinstance(grouped.indexes["group"], pd.MultiIndex) + keys = set(zip(grouped.data["dim_2"].values, grouped.data["b"].values)) + assert keys == set(pd.MultiIndex.from_frame(groups).values) + assert grouped.nterm == 3 - if use_fallback: - with pytest.raises(ValueError): - expr.groupby(groups).sum(use_fallback=use_fallback) - return - grouped = expr.groupby(groups).sum(use_fallback=use_fallback) - assert "group" in grouped.dims + +@pytest.mark.legacy +def test_linear_expression_groupby_with_dataframe_on_multiindex(u: Variable) -> None: + expr = 1 * u + n = len(u.data["dim_3"]) + groups = pd.DataFrame({"a": [1] * n}, index=u.indexes["dim_3"]) + with pytest.raises(ValueError): + expr.groupby(groups).sum(use_fallback=True) + with pytest.warns(LinopySemanticsWarning, match=r"stacked `group` MultiIndex"): + grouped = expr.groupby(groups).sum() assert isinstance(grouped.indexes["group"], pd.MultiIndex) - assert grouped.nterm == len_grouped_dim + assert grouped.nterm == n -@pytest.mark.parametrize("use_fallback", [True, False]) -def test_linear_expression_groupby_with_dataarray( - v: Variable, use_fallback: bool -) -> None: +@pytest.mark.v1 +def test_linear_expression_groupby_with_dataframe_on_multiindex_v1(u: Variable) -> None: + expr = 1 * u + n = len(u.data["dim_3"]) + groups = pd.DataFrame({"a": [1] * n}, index=u.indexes["dim_3"]) + grouped = expr.groupby(groups).sum() + assert not isinstance(grouped.indexes["group"], pd.MultiIndex) + assert set(grouped.data.a.values) == {1} + assert grouped.nterm == n + + +@pytest.mark.legacy +def test_linear_expression_groupby_with_dataarray(v: Variable) -> None: expr = 1 * v df = pd.DataFrame( {"a": [1] * 10 + [2] * 10, "b": list(range(4)) * 5}, index=v.indexes["dim_2"] ) groups = xr.DataArray(df) - # this should not be the case, see https://github.com/PyPSA/linopy/issues/351 - if use_fallback: - with pytest.raises((KeyError, IndexError)): - expr.groupby(groups).sum(use_fallback=use_fallback) - return + with pytest.raises((KeyError, IndexError)): + expr.groupby(groups).sum(use_fallback=True) + with pytest.warns(LinopySemanticsWarning, match=r"stacked `group` MultiIndex"): + grouped = expr.groupby(groups).sum() + assert set(grouped.data.group.values) == set(pd.MultiIndex.from_frame(df).values) + assert grouped.nterm == 3 - grouped = expr.groupby(groups).sum(use_fallback=use_fallback) - index = pd.MultiIndex.from_frame(df) - assert "group" in grouped.dims - assert set(grouped.data.group.values) == set(index.values) + +@pytest.mark.v1 +def test_linear_expression_groupby_with_dataarray_v1(v: Variable) -> None: + expr = 1 * v + df = pd.DataFrame( + {"a": [1] * 10 + [2] * 10, "b": list(range(4)) * 5}, index=v.indexes["dim_2"] + ) + grouped = expr.groupby(xr.DataArray(df)).sum() + assert not isinstance(grouped.indexes["group"], pd.MultiIndex) + keys = set(zip(grouped.data.a.values, grouped.data.b.values)) + assert keys == set(pd.MultiIndex.from_frame(df).values) assert grouped.nterm == 3 From 070b4bd06e86e6232468e32b7be43380cf8397ce Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Tue, 30 Jun 2026 22:24:20 +0200 Subject: [PATCH 30/33] test: make the MI `u` fixture semantics-aware for the v1 MI drop (#744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Toward v1 disallowing MultiIndex entirely: the shared `u` fixture now yields a (level1, level2) MultiIndex variable under legacy and the equivalent flat `dim_3` dim + level1/level2 aux coords under v1. Moved it out of the central `m` fixture so `m` is MI-free. The two v1 tests that asserted MI-specific behavior are rewritten to the flat+aux idiom: the level-projection "raises" test becomes explicit per-level weighting via the aux coord; the MI inner-join alignment test is marked legacy (flat-dim alignment is covered by the other align tests). Prep only — MI is still allowed under v1 at this point; no v1 test relies on it now. Suite green under both semantics. Co-Authored-By: Claude Opus 4.8 (1M context) --- test/conftest.py | 18 +++++++++++++++--- test/test_alignment.py | 3 ++- test/test_linear_expression.py | 10 +++++----- 3 files changed, 22 insertions(+), 9 deletions(-) diff --git a/test/conftest.py b/test/conftest.py index 6439168f..db50d359 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -117,9 +117,6 @@ def m() -> Model: m.add_variables(4, pd.Series([8, 10]), name="y") m.add_variables(0, pd.DataFrame([[1, 2], [3, 4], [5, 6]]).T, name="z") m.add_variables(coords=[pd.RangeIndex(20, name="dim_2")], name="v") - idx = pd.MultiIndex.from_product([[1, 2], ["a", "b"]], names=("level1", "level2")) - idx.name = "dim_3" - m.add_variables(coords=[idx], name="u") return m @@ -145,4 +142,19 @@ def v(m: Model) -> Variable: @pytest.fixture def u(m: Model) -> Variable: + """ + `dim_3` variable: a (level1, level2) MultiIndex under legacy, the flat dim + + level1/level2 aux coords equivalent under v1 (where MultiIndex is disallowed). + """ + from linopy.semantics import is_v1 + + if is_v1(): + m.add_variables(coords=[pd.RangeIndex(4, name="dim_3")], name="u") + return m.variables["u"].assign_coords( + level1=("dim_3", [1, 1, 2, 2]), + level2=("dim_3", ["a", "b", "a", "b"]), + ) + idx = pd.MultiIndex.from_product([[1, 2], ["a", "b"]], names=("level1", "level2")) + idx.name = "dim_3" + m.add_variables(coords=[idx], name="u") return m.variables["u"] diff --git a/test/test_alignment.py b/test/test_alignment.py index a5a850af..5d6971fc 100644 --- a/test/test_alignment.py +++ b/test/test_alignment.py @@ -1105,8 +1105,9 @@ def test_left_join_keeps_left_coords_and_fills(self, x: Variable) -> None: assert_varequal(x_obs, x) assert_equal(alpha_obs, DataArray([np.nan, 1], [[0, 1]])) + @pytest.mark.legacy def test_inner_join_over_multiindex(self, u: Variable) -> None: - """Inner join intersects MultiIndex coords element-wise across the stacked dim.""" + """Legacy: inner join intersects MultiIndex coords element-wise across the dim.""" beta = xr.DataArray( [1, 2, 3], [ diff --git a/test/test_linear_expression.py b/test/test_linear_expression.py index 1005152a..30f1c160 100644 --- a/test/test_linear_expression.py +++ b/test/test_linear_expression.py @@ -319,12 +319,12 @@ def test_multiply_expression_by_multiindex_level_constant(u: Variable) -> None: @pytest.mark.v1 -def test_multiply_expression_by_mi_level_constant_raises_v1(u: Variable) -> None: - """v1: the implicit level projection in arithmetic must be explicit.""" +def test_multiply_expression_by_level_aux_coord_v1(u: Variable) -> None: + """v1: weight each entry by its `level1` explicitly via the aux coord.""" by_level1 = xr.DataArray([10.0, 20.0], coords={"level1": [1, 2]}, dims=["level1"]) - - with pytest.raises(ValueError, match=r"not supported under the v1 convention"): - (1 * u) * by_level1 + weights = by_level1.sel(level1=u.coords["level1"]) + coeffs = ((1 * u) * weights).coeffs.squeeze("_term") + assert coeffs.values.tolist() == [10.0, 10.0, 20.0, 20.0] def test_linear_expression_with_errors(m: Model, x: Variable) -> None: From bb9b0292f24edf22b7b7fbfe73fbe516d94107f7 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Wed, 1 Jul 2026 08:08:56 +0200 Subject: [PATCH 31/33] feat(v1): disallow MultiIndex dim-coords entirely (#744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v1 now rejects a pandas MultiIndex dimension anywhere it would enter the model; legacy keeps it with a DeprecationWarning. This supersedes the branch's earlier "MI-allowed-but-strict" v1 design — under v1 a snapshot (or any dim) must be a flat dim with the levels as auxiliary coords (`reset_index`). enforce_no_multiindex(obj, *, context) scans an object's dim-indexes for a MultiIndex and raises (v1) / warns (legacy) with a message naming what the user did. It's wired at the caller-facing boundaries where MI can enter a persistent object, so the error is precise: - add_variables → context="variable 'x'" (catches coords and DataArray bounds) - add_constraints → context="constraint 'c'" (catches MI introduced via arithmetic) Tests: MultiIndex is now a legacy-only concept, so MI-building tests are marked legacy (or their fixtures skip under v1), and the obsolete @v1 tests that asserted the old MI-under-v1 semantics are removed and replaced by a v1 reject test. Full suite green under both semantics. Co-Authored-By: Claude Opus 4.8 (1M context) --- linopy/model.py | 9 +++++++++ linopy/semantics.py | 35 +++++++++++++++++++++++++++++++++ test/test_constraint.py | 17 ---------------- test/test_io.py | 4 ++++ test/test_legacy_violations.py | 36 +--------------------------------- test/test_linear_expression.py | 1 + test/test_variable.py | 20 ++++++++++--------- 7 files changed, 61 insertions(+), 61 deletions(-) diff --git a/linopy/model.py b/linopy/model.py index de5c089f..d4a3f985 100644 --- a/linopy/model.py +++ b/linopy/model.py @@ -841,6 +841,9 @@ def add_variables( if self.chunk: data = data.chunk(self.chunk) + from linopy.semantics import enforce_no_multiindex + + enforce_no_multiindex(data, context=f"variable {name!r}") variable = Variable(data, name=name, model=self, skip_broadcast=True) self.variables.add(variable) return variable @@ -1134,6 +1137,9 @@ def add_constraints( if self.chunk: data = data.chunk(self.chunk) + from linopy.semantics import enforce_no_multiindex + + enforce_no_multiindex(data, context=f"constraint {name!r}") constraint = Constraint(data, name=name, model=self, skip_broadcast=True) if freeze is None: freeze = self.freeze_constraints @@ -1203,6 +1209,9 @@ def add_indicator_constraints( data = self._allocate_constraint_labels(data, name) + from linopy.semantics import enforce_no_multiindex + + enforce_no_multiindex(data, context=f"constraint {name!r}") con = Constraint(data, name=name, model=self, skip_broadcast=True) freeze = self.freeze_constraints return self.constraints.add(con, freeze=freeze and not self.chunk) diff --git a/linopy/semantics.py b/linopy/semantics.py index 88f30b8f..918ccbde 100644 --- a/linopy/semantics.py +++ b/linopy/semantics.py @@ -232,6 +232,24 @@ def _legacy_group_multiindex_message(level_names: list[str]) -> str: ) +def _v1_multiindex_message(dim: str, context: str) -> str: + """A pandas MultiIndex dim-coord, rejected under v1.""" + return ( + f"{context} uses dimension {dim!r}, a `pd.MultiIndex`, which the v1 convention" + f" does not support. Decompose it to a flat dim with the levels as auxiliary" + f" coords first — e.g. `reset_index({dim!r})`." + ) + + +def _legacy_multiindex_message(dim: str, context: str) -> str: + """A pandas MultiIndex dim-coord, kept under legacy and rejected under v1.""" + return ( + f"{context} uses dimension {dim!r}, a `pd.MultiIndex`. Legacy keeps it as a" + f" first-class index; v1 rejects it — decompose it to a flat dim + level aux" + f" coords with `reset_index({dim!r})`." + _OPT_IN_HINT + ) + + _LINOPY_ROOT = os.path.dirname(os.path.abspath(__file__)) @@ -303,6 +321,23 @@ def enforce_aux_conflict(datasets: Sequence[Any], *, stacklevel: int = 5) -> Non warn_legacy(_legacy_aux_conflict_message(*conflict), stacklevel=stacklevel) +def enforce_no_multiindex( + obj: Any, *, context: str = "input", stacklevel: int = 4 +) -> None: + """Reject (v1) / deprecate (legacy) any MultiIndex dimension on ``obj``.""" + indexes = getattr(obj, "indexes", None) + if not indexes: + return + for dim, idx in indexes.items(): + if isinstance(idx, pd.MultiIndex): + if is_v1(): + raise ValueError(_v1_multiindex_message(str(dim), context)) + warn_legacy( + _legacy_multiindex_message(str(dim), context), stacklevel=stacklevel + ) + return + + def dim_coords_differ(a: DataArray, b: DataArray) -> bool: """True if a and b share a dimension whose coordinate labels disagree.""" return first_mismatched_dim(a, b) is not None diff --git a/test/test_constraint.py b/test/test_constraint.py index f74bddc7..7fe50a57 100644 --- a/test/test_constraint.py +++ b/test/test_constraint.py @@ -625,23 +625,6 @@ def test_constraint_rhs_setter_projects_multiindex_level() -> None: assert con.rhs.sel(dim_3=(2, "a")).item() == 20.0 -@pytest.mark.v1 -def test_constraint_rhs_setter_mi_level_raises_v1() -> None: - """v1: an rhs indexed by an MI level must be projected explicitly.""" - idx = pd.MultiIndex.from_product([[1, 2], ["a", "b"]], names=("level1", "level2")) - idx.name = "dim_3" - coords = xr.Coordinates.from_pandas_multiindex(idx, "dim_3") - m = Model() - x = m.add_variables(coords=coords, name="x") - con = m.add_constraints(1 * x >= 0, name="con") - - rhs_by_level = xr.DataArray( - [10.0, 20.0], coords={"level1": [1, 2]}, dims=["level1"] - ) - with pytest.raises(ValueError, match=r"not supported under the v1 convention"): - con.rhs = rhs_by_level - - def test_constraint_labels_setter_invalid(c: linopy.constraints.CSRConstraint) -> None: # Test that assigning labels raises AttributeError (Constraint is frozen) with pytest.raises(AttributeError): diff --git a/test/test_io.py b/test/test_io.py index 27cba396..40c7b0a5 100644 --- a/test/test_io.py +++ b/test/test_io.py @@ -57,6 +57,10 @@ def model_with_dash_names() -> Model: @pytest.fixture def model_with_multiindex() -> Model: + from linopy.semantics import is_v1 + + if is_v1(): + pytest.skip("v1 rejects MultiIndex; this model only builds under legacy") m = Model() index = pd.MultiIndex.from_tuples( diff --git a/test/test_legacy_violations.py b/test/test_legacy_violations.py index 0247571d..8ad4decb 100644 --- a/test/test_legacy_violations.py +++ b/test/test_legacy_violations.py @@ -854,26 +854,6 @@ def test_quadratic_merge_reordered_aligns(self, m: Model) -> None: labels = {int(v) for v in result.vars.sel(e="x").values.ravel() if v >= 0} assert labels == {int(x.labels.sel(e="x")), int(y.labels.sel(e="x"))} - @pytest.mark.v1 - def test_reordered_multiindex_aligns_by_tuple(self, m: Model) -> None: - mi1 = pd.MultiIndex.from_tuples( - [(1, "a"), (1, "b"), (2, "a")], names=["p", "s"] - ) - mi2 = pd.MultiIndex.from_tuples( - [(2, "a"), (1, "b"), (1, "a")], names=["p", "s"] - ) - mi1.name = "snap" - mi2.name = "snap" - x = m.add_variables(coords=[mi1], name="x") + pd.Series( - [1.0, 2.0, 3.0], index=mi1 - ) - y = m.add_variables(coords=[mi2], name="y") + pd.Series( - [10.0, 20.0, 30.0], index=mi2 - ) - result = x + y - got = dict(zip(map(tuple, result.indexes["snap"]), result.const.values)) - assert got == {(1, "a"): 31.0, (1, "b"): 22.0, (2, "a"): 13.0} - @pytest.mark.legacy def test_reordered_merge_positional_legacy(self, m: Model) -> None: ea = pd.Index(["costs", "penalty"], name="e") @@ -942,21 +922,6 @@ def test_quadratic_merge_reordered_pairs_positionally_legacy( labels = {int(v) for v in result.vars.sel(e="x").values.ravel() if v >= 0} assert labels == {int(x.labels.sel(e="x")), int(y.labels.sel(e="z"))} - @pytest.mark.v1 - def test_different_multiindex_raises_dim_mismatch(self, m: Model) -> None: - mi1 = pd.MultiIndex.from_tuples( - [(1, "a"), (1, "b"), (2, "a")], names=["p", "s"] - ) - mi2 = pd.MultiIndex.from_tuples( - [(1, "a"), (1, "b"), (3, "c")], names=["p", "s"] - ) - mi1.name = "snap" - mi2.name = "snap" - x = m.add_variables(coords=[mi1], name="x") - y = m.add_variables(coords=[mi2], name="y") - with pytest.raises(ValueError, match="shared dimension 'snap'"): - (1 * x) + (1 * y) - @pytest.mark.legacy def test_var_plus_var_different_labels_silent( self, x: Variable, x_other: Variable @@ -1428,6 +1393,7 @@ def test_where_creates_absence(self, x: Variable) -> None: assert (masked.labels.values[2:] == -1).all() assert not (masked.labels.values[:2] == -1).any() + @pytest.mark.legacy def test_unstack_creates_absence_at_missing_combinations(self, m: Model) -> None: """ §4 — ``.unstack`` of a non-rectangular MultiIndex leaves the diff --git a/test/test_linear_expression.py b/test/test_linear_expression.py index 30f1c160..48aedb51 100644 --- a/test/test_linear_expression.py +++ b/test/test_linear_expression.py @@ -1685,6 +1685,7 @@ def test_multi_key_dataarrays_unsupported( ("timestep", ["t1", "t2", "t3"], [[0, 3], [1, 4], [2, 5]]), ], ) + @pytest.mark.legacy def test_multiindex_level( self, level: str, values: list, vars_: list, use_fallback: bool ) -> None: diff --git a/test/test_variable.py b/test/test_variable.py index 80883c3c..ee056674 100644 --- a/test/test_variable.py +++ b/test/test_variable.py @@ -875,8 +875,9 @@ def test_string_coords_mismatch(self, model: "Model") -> None: ) +@pytest.mark.legacy class TestAddVariablesMultiIndexCoords: - """MultiIndex-specific coord handling in add_variables.""" + """MultiIndex-specific coord handling in add_variables (legacy; v1 rejects MI).""" @pytest.fixture def model(self) -> "Model": @@ -939,14 +940,6 @@ def test_single_level_bound_broadcasts( assert var.dims == ("multi",) assert (var.data.upper == [5, 5, 6, 6]).all() - @pytest.mark.v1 - def test_single_level_bound_raises_v1( - self, model: "Model", midx: pd.MultiIndex - ) -> None: - bound = DataArray([5, 6], dims=["l1"], coords={"l1": [0, 1]}) - with pytest.raises(ValueError, match=r"not supported under the v1 convention"): - model.add_variables(upper=bound, coords=[midx], name="x") - def test_incomplete_level_bound_raises( self, model: "Model", midx: pd.MultiIndex ) -> None: @@ -954,3 +947,12 @@ def test_incomplete_level_bound_raises( bound = pd.Series([1, 2], index=subset) with pytest.raises(ValueError, match="no value for .* level combination"): model.add_variables(upper=bound, coords=[midx], name="x") + + +@pytest.mark.v1 +def test_add_variables_multiindex_rejected_v1() -> None: + """v1: an MI dim-coord is rejected at construction with a reset_index hint.""" + mi = pd.MultiIndex.from_product([[0, 1], ["a", "b"]], names=("l1", "l2")) + mi.name = "multi" + with pytest.raises(ValueError, match=r"v1 convention does not support"): + Model().add_variables(lower=0, upper=1, coords=[mi], name="x") From 94a8a3cfb45a88d269e0604f4e9a907f07bdb235 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Wed, 1 Jul 2026 08:20:37 +0200 Subject: [PATCH 32/33] feat(v1): guard add_objective against MultiIndex; hoist imports (#744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit add_objective assigns the expression directly (bypassing Objective.__init__), so add the enforce_no_multiindex check there too — an MI forced onto an expression and handed to the objective now rejects under v1 with the "objective ..." context. Covered by a v1 reject test alongside the add_variables/add_constraints ones. Also move the `enforce_no_multiindex` import to module level in model.py (no import cycle — semantics imports none of model/variables/constraints/objective). Co-Authored-By: Claude Opus 4.8 (1M context) --- linopy/model.py | 8 ++------ test/test_variable.py | 13 +++++++++++++ 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/linopy/model.py b/linopy/model.py index d4a3f985..dd603d52 100644 --- a/linopy/model.py +++ b/linopy/model.py @@ -79,6 +79,7 @@ add_piecewise_formulation, ) from linopy.remote import RemoteHandler +from linopy.semantics import enforce_no_multiindex try: from linopy.remote import OetcHandler @@ -841,8 +842,6 @@ def add_variables( if self.chunk: data = data.chunk(self.chunk) - from linopy.semantics import enforce_no_multiindex - enforce_no_multiindex(data, context=f"variable {name!r}") variable = Variable(data, name=name, model=self, skip_broadcast=True) self.variables.add(variable) @@ -1137,8 +1136,6 @@ def add_constraints( if self.chunk: data = data.chunk(self.chunk) - from linopy.semantics import enforce_no_multiindex - enforce_no_multiindex(data, context=f"constraint {name!r}") constraint = Constraint(data, name=name, model=self, skip_broadcast=True) if freeze is None: @@ -1209,8 +1206,6 @@ def add_indicator_constraints( data = self._allocate_constraint_labels(data, name) - from linopy.semantics import enforce_no_multiindex - enforce_no_multiindex(data, context=f"constraint {name!r}") con = Constraint(data, name=name, model=self, skip_broadcast=True) freeze = self.freeze_constraints @@ -1253,6 +1248,7 @@ def add_objective( ) if isinstance(expr, Variable): expr = 1 * expr + enforce_no_multiindex(expr, context="objective") self.objective.expression = expr self.objective.sense = sense diff --git a/test/test_variable.py b/test/test_variable.py index ee056674..680e6bbf 100644 --- a/test/test_variable.py +++ b/test/test_variable.py @@ -956,3 +956,16 @@ def test_add_variables_multiindex_rejected_v1() -> None: mi.name = "multi" with pytest.raises(ValueError, match=r"v1 convention does not support"): Model().add_variables(lower=0, upper=1, coords=[mi], name="x") + + +@pytest.mark.v1 +def test_add_constraint_and_objective_multiindex_rejected_v1() -> None: + """v1: an MI forced onto an expression is rejected at the constraint/objective.""" + mi = pd.MultiIndex.from_product([[0, 1], ["a", "b"]], names=("l1", "l2")) + mi.name = "d" + m = Model() + x = m.add_variables(coords=[pd.RangeIndex(4, name="d")], name="x") + with pytest.raises(ValueError, match=r"constraint .* v1 convention does not"): + m.add_constraints((1 * x).assign_coords(d=mi) >= 0, name="c") + with pytest.raises(ValueError, match=r"objective .* v1 convention does not"): + m.add_objective((1 * x).assign_coords(d=mi)) From e87efc1f42bfdd64643aa347eb920179afb91abd Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Wed, 1 Jul 2026 10:40:28 +0200 Subject: [PATCH 33/33] docs(arithmetics): track open-items.md; mark #744 resolved (#744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Commit the v1-rollout checklist (previously untracked) as the shared status tracker, refreshed to the current state: the one open design decision (#744, MultiIndex storage) is resolved — v1 disallows MI — and the remaining work is organised by goals.md's three stages (opt-in / default / 1.0). The transition-surface-complete + changelog items are placed at stage 1, matching goals.md step 1 (warn on legacy, raise on v1). Co-Authored-By: Claude Opus 4.8 (1M context) --- arithmetics-design/open-items.md | 36 +++++++++++++++++++------------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/arithmetics-design/open-items.md b/arithmetics-design/open-items.md index 2c94d407..984036e1 100644 --- a/arithmetics-design/open-items.md +++ b/arithmetics-design/open-items.md @@ -6,26 +6,29 @@ coordinate-alignment intro rules all have a v1 implementation. The high-level schedule lives in [`goals.md`](goals.md) (opt-in → default → 1.0); this file tracks the concrete items per stage. -One open **design** decision remains — [#744] MultiIndex storage. No open -arithmetic-rule questions. +The one open **design** decision — [#744] MultiIndex storage — is **resolved**: +v1 disallows first-class `pd.MultiIndex` (a flat dim + auxiliary level coords), +implemented in [#803]. No open arithmetic-rule questions remain; everything left +is rollout + cleanup. ## Stage 1 — release v1 (opt-in) Legacy stays the default. The transition surface must already be **complete** here ([`goals.md`] step 1: *warn on legacy, raise on v1*). -- [ ] **Resolve [#744] — MultiIndex storage — before #717 merges.** First-class - `pd.MultiIndex` vs. a flat dim + auxiliary level coords. §11's - stacked-MultiIndex rule and the storage of a `snapshot`-style - `(period, timestep)` dim both depend on it — and §11 **ships in #717**, so - changing the model after v1 is released would change observable §11 behaviour. - Must be locked (and its implementation in the same release cut) before #717 - ships v1, not deferred to the default flip. +- [x] **Resolve [#744] — MultiIndex storage — before #717 merges.** Resolved: + v1 disallows first-class `pd.MultiIndex` (flat dim + auxiliary level coords). + §11's stacked-MultiIndex rule and the storage of a `snapshot`-style + `(period, timestep)` dim both depend on this, and §11 **ships in #717** — so + the implementation ([#803]) must land in the **same release cut** as #717, + else released §11 behaviour would change when it lands. - [ ] Land [#717] (v1 semantics) → `master` +- [ ] Land [#803] (v1 MultiIndex drop) with it — same cut, so §11 ships final - [ ] **Transition surface complete** — every behaviour-change site raises under v1 *and* warns under legacy (`warn_legacy`, naming the fix). This is the "no silent change" guarantee ([`goals.md`] transitioning goal #3): shipping v1 - with a gap would silently change any model that opts in. + with a gap would silently change any model that opts in. Includes #803's + MultiIndex rejects and the multi-key-`groupby`-flat change as new fork sites. - [ ] Changelog note — v1 available via `options['semantics'] = 'v1'`; legacy remains the default; link [`convention.md`]. @@ -40,8 +43,9 @@ here ([`goals.md`] step 1: *warn on legacy, raise on v1*). - [ ] The **strip**: the concentrated MI machinery (the `alignment.py` level-projection subsystem, netcdf MI (de)serialize) *and* the scattered surface (`assign_multiindex_safe` ×39, `isinstance(MultiIndex)` guards, MI - branches) — plus every other `grep "LEGACY: remove at 1.0"` marker. - Dependency-ordered checklist in [`legacy-removal.md`]. + branches) — all kept live for legacy until now — plus every other + `grep "LEGACY: remove at 1.0"` marker. Dependency-ordered checklist in + [`legacy-removal.md`]. - [ ] Reframe [`convention.md`] / [`goals.md`] — drop the "v1"/legacy framing once there is only one convention. @@ -55,8 +59,11 @@ here ([`goals.md`] step 1: *warn on legacy, raise on v1*). ## Decisions -- [ ] **[#744] — MultiIndex storage** — the one open design decision; gates the - **v1 release** (§11 ships in #717 and depends on it), not just the default flip. +- [x] **[#744] — MultiIndex storage** → v1 disallows MI (flat + aux coords), + implemented in [#803]; must land in the same release cut as #717 (§11 depends + on it). +- [x] **Legacy warnings live from the opt-in release** → yes; the transition + surface is complete at stage 1, not deferred. - [ ] **When v1 becomes the default** — pick the release; gated on the migration guide. @@ -64,6 +71,7 @@ here ([`goals.md`] step 1: *warn on legacy, raise on v1*). [#744]: https://github.com/PyPSA/linopy/issues/744 [#714]: https://github.com/PyPSA/linopy/issues/714 [#717]: https://github.com/PyPSA/linopy/pull/717 +[#803]: https://github.com/PyPSA/linopy/pull/803 [`goals.md`]: goals.md [`docs-plan.md`]: docs-plan.md [`legacy-removal.md`]: legacy-removal.md