feat(v1): drop first-class MultiIndex — feasibility + implementation (#744)

arithmetics-design/open-items.md

@@ -6,26 +6,29 @@ coordinate-alignment intro rules all have a v1 implementation. The high-level
 schedule lives in [`goals.md`](goals.md) (opt-in → default → 1.0); this file
 tracks the concrete items per stage.
 
-One open **design** decision remains — [#744] MultiIndex storage. No open
-arithmetic-rule questions.
+The one open **design** decision — [#744] MultiIndex storage — is **resolved**:
+v1 disallows first-class `pd.MultiIndex` (a flat dim + auxiliary level coords),
+implemented in [#803]. No open arithmetic-rule questions remain; everything left
+is rollout + cleanup.
 
 ## Stage 1 — release v1 (opt-in)
 
 Legacy stays the default. The transition surface must already be **complete**
 here ([`goals.md`] step 1: *warn on legacy, raise on v1*).
 
-- [ ] **Resolve [#744] — MultiIndex storage — before #717 merges.** First-class
-  `pd.MultiIndex` vs. a flat dim + auxiliary level coords. §11's
-  stacked-MultiIndex rule and the storage of a `snapshot`-style
-  `(period, timestep)` dim both depend on it — and §11 **ships in #717**, so
-  changing the model after v1 is released would change observable §11 behaviour.
-  Must be locked (and its implementation in the same release cut) before #717
-  ships v1, not deferred to the default flip.
+- [x] **Resolve [#744] — MultiIndex storage — before #717 merges.** Resolved:
+  v1 disallows first-class `pd.MultiIndex` (flat dim + auxiliary level coords).
+  §11's stacked-MultiIndex rule and the storage of a `snapshot`-style
+  `(period, timestep)` dim both depend on this, and §11 **ships in #717** — so
+  the implementation ([#803]) must land in the **same release cut** as #717,
+  else released §11 behaviour would change when it lands.
 - [ ] Land [#717] (v1 semantics) → `master`
+- [ ] Land [#803] (v1 MultiIndex drop) with it — same cut, so §11 ships final
 - [ ] **Transition surface complete** — every behaviour-change site raises under
   v1 *and* warns under legacy (`warn_legacy`, naming the fix). This is the
   "no silent change" guarantee ([`goals.md`] transitioning goal #3): shipping v1
-  with a gap would silently change any model that opts in.
+  with a gap would silently change any model that opts in. Includes #803's
+  MultiIndex rejects and the multi-key-`groupby`-flat change as new fork sites.
 - [ ] Changelog note — v1 available via `options['semantics'] = 'v1'`; legacy
   remains the default; link [`convention.md`].
 
@@ -40,8 +43,9 @@ here ([`goals.md`] step 1: *warn on legacy, raise on v1*).
 - [ ] The **strip**: the concentrated MI machinery (the `alignment.py`
   level-projection subsystem, netcdf MI (de)serialize) *and* the scattered
   surface (`assign_multiindex_safe` ×39, `isinstance(MultiIndex)` guards, MI
-  branches) — plus every other `grep "LEGACY: remove at 1.0"` marker.
-  Dependency-ordered checklist in [`legacy-removal.md`].
+  branches) — all kept live for legacy until now — plus every other
+  `grep "LEGACY: remove at 1.0"` marker. Dependency-ordered checklist in
+  [`legacy-removal.md`].
 - [ ] Reframe [`convention.md`] / [`goals.md`] — drop the "v1"/legacy framing
   once there is only one convention.
 
@@ -55,15 +59,19 @@ here ([`goals.md`] step 1: *warn on legacy, raise on v1*).
 
 ## Decisions
 
-- [ ] **[#744] — MultiIndex storage** — the one open design decision; gates the
-  **v1 release** (§11 ships in #717 and depends on it), not just the default flip.
+- [x] **[#744] — MultiIndex storage** → v1 disallows MI (flat + aux coords),
+  implemented in [#803]; must land in the same release cut as #717 (§11 depends
+  on it).
+- [x] **Legacy warnings live from the opt-in release** → yes; the transition
+  surface is complete at stage 1, not deferred.
 - [ ] **When v1 becomes the default** — pick the release; gated on the migration
   guide.
 
 <!-- references -->
 [#744]: https://github.com/PyPSA/linopy/issues/744
 [#714]: https://github.com/PyPSA/linopy/issues/714
 [#717]: https://github.com/PyPSA/linopy/pull/717
+[#803]: https://github.com/PyPSA/linopy/pull/803
 [`goals.md`]: goals.md
 [`docs-plan.md`]: docs-plan.md
 [`legacy-removal.md`]: legacy-removal.md

linopy/expressions.py

@@ -91,6 +91,7 @@
 from linopy.semantics import (
     _legacy_coord_mismatch_message,
     _legacy_coord_reorder_message,
+    _legacy_group_multiindex_message,
     _legacy_nan_rhs_constraint_message,
     _shared_dim_mismatch_message,
     absorb_absence,
@@ -378,10 +379,19 @@ def sum(
 
             if int_map is not None:
                 index = ds.indexes[GROUP_DIM].map({v: k for k, v in int_map.items()})
-                index.names = [str(col) for col in orig_group.columns]
+                level_names = [str(col) for col in orig_group.columns]
+                index.names = level_names
                 index.name = GROUP_DIM
-                new_coords = Coordinates.from_pandas_multiindex(index, GROUP_DIM)
-                ds = ds.assign_coords(new_coords)
+                stacked_survives = multikey_frame is None or observed
+                if stacked_survives and is_v1():  # flat group dim + keys as aux coords
+                    ds = ds.assign_coords(
+                        {n: (GROUP_DIM, index.get_level_values(n)) for n in level_names}
+                    )
+                else:
+                    if multikey_frame is None:  # user-facing frame grouper
+                        warn_legacy(_legacy_group_multiindex_message(level_names))
+                    new_coords = Coordinates.from_pandas_multiindex(index, GROUP_DIM)
+                    ds = ds.assign_coords(new_coords)
 
             ds = ds.rename({GROUP_DIM: final_group_name})
             if multikey_frame is not None and not observed:

linopy/model.py

@@ -79,6 +79,7 @@
     add_piecewise_formulation,
 )
 from linopy.remote import RemoteHandler
+from linopy.semantics import enforce_no_multiindex
 
 try:
     from linopy.remote import OetcHandler
@@ -841,6 +842,7 @@ def add_variables(
         if self.chunk:
             data = data.chunk(self.chunk)
 
+        enforce_no_multiindex(data, context=f"variable {name!r}")
         variable = Variable(data, name=name, model=self, skip_broadcast=True)
         self.variables.add(variable)
         return variable
@@ -1134,6 +1136,7 @@ def add_constraints(
         if self.chunk:
             data = data.chunk(self.chunk)
 
+        enforce_no_multiindex(data, context=f"constraint {name!r}")
         constraint = Constraint(data, name=name, model=self, skip_broadcast=True)
         if freeze is None:
             freeze = self.freeze_constraints
@@ -1203,6 +1206,7 @@ def add_indicator_constraints(
 
         data = self._allocate_constraint_labels(data, name)
 
+        enforce_no_multiindex(data, context=f"constraint {name!r}")
         con = Constraint(data, name=name, model=self, skip_broadcast=True)
         freeze = self.freeze_constraints
         return self.constraints.add(con, freeze=freeze and not self.chunk)
@@ -1244,6 +1248,7 @@ def add_objective(
             )
         if isinstance(expr, Variable):
             expr = 1 * expr
+        enforce_no_multiindex(expr, context="objective")
         self.objective.expression = expr
         self.objective.sense = sense
 

linopy/semantics.py

@@ -217,6 +217,39 @@ def _legacy_masked_variable_message(name: str) -> str:
     )
 
 
+def _legacy_group_multiindex_message(level_names: list[str]) -> str:
+    """A multi-key groupby returns a stacked ``group`` MultiIndex under legacy."""
+    levels = ", ".join(repr(n) for n in level_names)
+    return (
+        f"A groupby with a frame grouper returned a stacked `group` MultiIndex over"
+        f" ({levels})."
+        " Under legacy the `group` dim is that MultiIndex, so `.sel(group=(...))` by"
+        " tuple works. Under v1 the `group` dim is flat with those keys as ordinary"
+        f" aux coords instead — select via `.groupby`/`.where` on {levels}, not a"
+        " level tuple."
+        f"\n  Resolve:   replace `.sel(group=(...))` with selection on the"
+        f" {levels} aux coord(s)." + _OPT_IN_HINT
+    )
+
+
+def _v1_multiindex_message(dim: str, context: str) -> str:
+    """A pandas MultiIndex dim-coord, rejected under v1."""
+    return (
+        f"{context} uses dimension {dim!r}, a `pd.MultiIndex`, which the v1 convention"
+        f" does not support. Decompose it to a flat dim with the levels as auxiliary"
+        f" coords first — e.g. `reset_index({dim!r})`."
+    )
+
+
+def _legacy_multiindex_message(dim: str, context: str) -> str:
+    """A pandas MultiIndex dim-coord, kept under legacy and rejected under v1."""
+    return (
+        f"{context} uses dimension {dim!r}, a `pd.MultiIndex`. Legacy keeps it as a"
+        f" first-class index; v1 rejects it — decompose it to a flat dim + level aux"
+        f" coords with `reset_index({dim!r})`." + _OPT_IN_HINT
+    )
+
+
 _LINOPY_ROOT = os.path.dirname(os.path.abspath(__file__))
 
 
@@ -288,6 +321,23 @@ def enforce_aux_conflict(datasets: Sequence[Any], *, stacklevel: int = 5) -> Non
     warn_legacy(_legacy_aux_conflict_message(*conflict), stacklevel=stacklevel)
 
 
+def enforce_no_multiindex(
+    obj: Any, *, context: str = "input", stacklevel: int = 4
+) -> None:
+    """Reject (v1) / deprecate (legacy) any MultiIndex dimension on ``obj``."""
+    indexes = getattr(obj, "indexes", None)
+    if not indexes:
+        return
+    for dim, idx in indexes.items():
+        if isinstance(idx, pd.MultiIndex):
+            if is_v1():
+                raise ValueError(_v1_multiindex_message(str(dim), context))
+            warn_legacy(
+                _legacy_multiindex_message(str(dim), context), stacklevel=stacklevel
+            )
+            return
+
+
 def dim_coords_differ(a: DataArray, b: DataArray) -> bool:
     """True if a and b share a dimension whose coordinate labels disagree."""
     return first_mismatched_dim(a, b) is not None

test/conftest.py

@@ -117,9 +117,6 @@ def m() -> Model:
     m.add_variables(4, pd.Series([8, 10]), name="y")
     m.add_variables(0, pd.DataFrame([[1, 2], [3, 4], [5, 6]]).T, name="z")
     m.add_variables(coords=[pd.RangeIndex(20, name="dim_2")], name="v")
-    idx = pd.MultiIndex.from_product([[1, 2], ["a", "b"]], names=("level1", "level2"))
-    idx.name = "dim_3"
-    m.add_variables(coords=[idx], name="u")
     return m
 
 
@@ -145,4 +142,19 @@ def v(m: Model) -> Variable:
 
 @pytest.fixture
 def u(m: Model) -> Variable:
+    """
+    `dim_3` variable: a (level1, level2) MultiIndex under legacy, the flat dim +
+    level1/level2 aux coords equivalent under v1 (where MultiIndex is disallowed).
+    """
+    from linopy.semantics import is_v1
+
+    if is_v1():
+        m.add_variables(coords=[pd.RangeIndex(4, name="dim_3")], name="u")
+        return m.variables["u"].assign_coords(
+            level1=("dim_3", [1, 1, 2, 2]),
+            level2=("dim_3", ["a", "b", "a", "b"]),
+        )
+    idx = pd.MultiIndex.from_product([[1, 2], ["a", "b"]], names=("level1", "level2"))
+    idx.name = "dim_3"
+    m.add_variables(coords=[idx], name="u")
     return m.variables["u"]

test/test_alignment.py

@@ -1105,8 +1105,9 @@ def test_left_join_keeps_left_coords_and_fills(self, x: Variable) -> None:
         assert_varequal(x_obs, x)
         assert_equal(alpha_obs, DataArray([np.nan, 1], [[0, 1]]))
 
+    @pytest.mark.legacy
     def test_inner_join_over_multiindex(self, u: Variable) -> None:
-        """Inner join intersects MultiIndex coords element-wise across the stacked dim."""
+        """Legacy: inner join intersects MultiIndex coords element-wise across the dim."""
         beta = xr.DataArray(
             [1, 2, 3],
             [

test/test_constraint.py

@@ -625,23 +625,6 @@ def test_constraint_rhs_setter_projects_multiindex_level() -> None:
     assert con.rhs.sel(dim_3=(2, "a")).item() == 20.0
 
 
-@pytest.mark.v1
-def test_constraint_rhs_setter_mi_level_raises_v1() -> None:
-    """v1: an rhs indexed by an MI level must be projected explicitly."""
-    idx = pd.MultiIndex.from_product([[1, 2], ["a", "b"]], names=("level1", "level2"))
-    idx.name = "dim_3"
-    coords = xr.Coordinates.from_pandas_multiindex(idx, "dim_3")
-    m = Model()
-    x = m.add_variables(coords=coords, name="x")
-    con = m.add_constraints(1 * x >= 0, name="con")
-
-    rhs_by_level = xr.DataArray(
-        [10.0, 20.0], coords={"level1": [1, 2]}, dims=["level1"]
-    )
-    with pytest.raises(ValueError, match=r"not supported under the v1 convention"):
-        con.rhs = rhs_by_level
-
-
 def test_constraint_labels_setter_invalid(c: linopy.constraints.CSRConstraint) -> None:
     # Test that assigning labels raises AttributeError (Constraint is frozen)
     with pytest.raises(AttributeError):

test/test_io.py

@@ -57,6 +57,10 @@ def model_with_dash_names() -> Model:
 
 @pytest.fixture
 def model_with_multiindex() -> Model:
+    from linopy.semantics import is_v1
+
+    if is_v1():
+        pytest.skip("v1 rejects MultiIndex; this model only builds under legacy")
     m = Model()
 
     index = pd.MultiIndex.from_tuples(

test/test_legacy_violations.py

@@ -854,26 +854,6 @@ def test_quadratic_merge_reordered_aligns(self, m: Model) -> None:
         labels = {int(v) for v in result.vars.sel(e="x").values.ravel() if v >= 0}
         assert labels == {int(x.labels.sel(e="x")), int(y.labels.sel(e="x"))}
 
-    @pytest.mark.v1
-    def test_reordered_multiindex_aligns_by_tuple(self, m: Model) -> None:
-        mi1 = pd.MultiIndex.from_tuples(
-            [(1, "a"), (1, "b"), (2, "a")], names=["p", "s"]
-        )
-        mi2 = pd.MultiIndex.from_tuples(
-            [(2, "a"), (1, "b"), (1, "a")], names=["p", "s"]
-        )
-        mi1.name = "snap"
-        mi2.name = "snap"
-        x = m.add_variables(coords=[mi1], name="x") + pd.Series(
-            [1.0, 2.0, 3.0], index=mi1
-        )
-        y = m.add_variables(coords=[mi2], name="y") + pd.Series(
-            [10.0, 20.0, 30.0], index=mi2
-        )
-        result = x + y
-        got = dict(zip(map(tuple, result.indexes["snap"]), result.const.values))
-        assert got == {(1, "a"): 31.0, (1, "b"): 22.0, (2, "a"): 13.0}
-
     @pytest.mark.legacy
     def test_reordered_merge_positional_legacy(self, m: Model) -> None:
         ea = pd.Index(["costs", "penalty"], name="e")
@@ -942,21 +922,6 @@ def test_quadratic_merge_reordered_pairs_positionally_legacy(
         labels = {int(v) for v in result.vars.sel(e="x").values.ravel() if v >= 0}
         assert labels == {int(x.labels.sel(e="x")), int(y.labels.sel(e="z"))}
 
-    @pytest.mark.v1
-    def test_different_multiindex_raises_dim_mismatch(self, m: Model) -> None:
-        mi1 = pd.MultiIndex.from_tuples(
-            [(1, "a"), (1, "b"), (2, "a")], names=["p", "s"]
-        )
-        mi2 = pd.MultiIndex.from_tuples(
-            [(1, "a"), (1, "b"), (3, "c")], names=["p", "s"]
-        )
-        mi1.name = "snap"
-        mi2.name = "snap"
-        x = m.add_variables(coords=[mi1], name="x")
-        y = m.add_variables(coords=[mi2], name="y")
-        with pytest.raises(ValueError, match="shared dimension 'snap'"):
-            (1 * x) + (1 * y)
-
     @pytest.mark.legacy
     def test_var_plus_var_different_labels_silent(
         self, x: Variable, x_other: Variable
@@ -1428,6 +1393,7 @@ def test_where_creates_absence(self, x: Variable) -> None:
         assert (masked.labels.values[2:] == -1).all()
         assert not (masked.labels.values[:2] == -1).any()
 
+    @pytest.mark.legacy
     def test_unstack_creates_absence_at_missing_combinations(self, m: Model) -> None:
         """
         §4 — ``.unstack`` of a non-rectangular MultiIndex leaves the