Superseded by #85: Investigate HMM backend surface

.github/workflows/test-and-lint.yml

@@ -23,11 +23,7 @@ jobs:
           cache-dependency-glob: "uv.lock"
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies (all extras)
-        if: ${{ matrix.python-version == '3.9' }}
         run: uv sync --group dev --all-extras
-      - name: Install dependencies (stats extra)
-        if: ${{ matrix.python-version != '3.9' }}
-        run: uv sync --group dev --extra stats
       - name: Testing
         run: uv run poe test
       - name: Upload coverage reports to Codecov

CHANGELOG.md

@@ -8,6 +8,19 @@ For more information see the
 [Github Releases Page](https://github.com/mad-lab-fau/gaitmap/releases) of this 
 project.
 
+## [Unreleased]
+
+### Scientific Changes
+
+- Repaired the shipped pretrained Roth HMM artifact so its compiled global `end_probs` now represent the intended
+  sequence semantics: uniform stopping probability across the five transition states and zero stopping probability
+  within stride states. This removes the invalid all-zero terminal distribution from the serialized model and aligns
+  backend-agnostic inference with the intended model semantics.
+- Fixed the legacy `pomegranate 0.14` fused HMM path to preserve sequence-end semantics when composing the final model.
+  The fused model now estimates global end probabilities from labeled sequence endings and normalizes them together
+  with outgoing transitions instead of dropping terminal probabilities during composition. This can slightly shift
+  decoded stride boundaries near sequence tails. (Issue: https://github.com/mad-lab-fau/gaitmap/issues/80)
+
 ## [2.6.0] - 2026-03-05
 
 ### Scientific Changes

_tasks.py

@@ -44,7 +44,7 @@ def update_version(version) -> None:
         .stdout.decode()
         .strip()
     )
-    update_version_strings(HERE / "gaitmap/__init__.py", new_version)
+    update_version_strings(HERE / "src/gaitmap/__init__.py", new_version)
     # Update the gaitmap_mad version as well
     subprocess.run(
         ["uv", "version", new_version, "--project", str(HERE / "packages/gaitmap_mad"), "--frozen"],

docs/conf.py

@@ -219,8 +219,7 @@ def get_nested_attr(obj, attr):
     new_obj = getattr(obj, attrs[0])
     if len(attrs) == 1:
         return new_obj
-    else:
-        return get_nested_attr(new_obj, attrs[1])
+    return get_nested_attr(new_obj, attrs[1])
 
 
 linkcode_resolve = make_linkcode_resolve(

examples/datasets_and_pipelines/cross_validation.py

@@ -30,7 +30,6 @@
 import pandas as pd
 from tpcp import CloneFactory, Dataset, OptimizableParameter, OptimizablePipeline, Parameter
 
-
 from gaitmap.data_transform import TrainableAbsMaxScaler
 from gaitmap.example_data import get_healthy_example_imu_data, get_healthy_example_stride_borders
 from gaitmap.stride_segmentation import (

examples/datasets_and_pipelines/gridsearch.py

@@ -12,7 +12,6 @@
 
 import pandas as pd
 
-
 # %%
 # To perform a GridSearch (or any other form of parameter optimization in Gaitmap), we first need to have a
 # **Dataset**, a **Pipeline** and a **score** function.

examples/datasets_and_pipelines/gridsearch_cv.py

@@ -25,7 +25,6 @@
 import numpy as np
 import pandas as pd
 
-
 from gaitmap.data_transform import TrainableAbsMaxScaler
 from gaitmap.utils.array_handling import iterate_region_data
 

examples/stride_segmentation/roth_hmm_stride_segmentation.py

@@ -55,10 +55,13 @@
 
 roth_hmm_model = PreTrainedRothSegmentationModel()
 
-print(f"Number of states, stride-model: {roth_hmm_model.stride_model.n_states:d}")
-print(f"Number of states, transition-model: {roth_hmm_model.transition_model.n_states:d}")
+stride_model = roth_hmm_model.model_config.get_module("stride")
+transition_model = roth_hmm_model.model_config.transition_model
+
+print(f"Number of states, stride-model: {stride_model.n_states:d}")
+print(f"Number of states, transition-model: {transition_model.n_states:d}")
 np.set_printoptions(precision=3, linewidth=180, suppress=True)
-print(f"Transition matrix:\n{roth_hmm_model.model.dense_transition_matrix()[0:-2, 0:-2]}")
+print(f"Transition matrix:\n{roth_hmm_model.model.compiled.graph.transition_probs}")
 
 # %%
 # Predicting hidden states / Stride borders

examples/stride_segmentation/segmentation_hmm_training.py

@@ -5,7 +5,7 @@
 ==========================
 
 This example illustrates how a Hidden Markov Model (HMM) implemented by the
-:class:`~gaitmap.stride_segmentation.hmm.RothSegmentationHmm` can be trained from IMU data and presegmented stride lists.
+:class:`~gaitmap.stride_segmentation.hmm.RothSegmentationHmm` can be trained from IMU data and typed region lists.
 The used implementation is based on the work of Roth et al [1]_
 
 .. [1] Roth, N., Küderle, A., Ullrich, M., Gladow, T., Marxreiter F., Klucken, J., Eskofier, B. & Kluge F. (2021).
@@ -84,28 +84,33 @@
 # different in architecture, number of states or number of gaussian mixture model (GMM) components.
 # In this example all configurable parameters are exposed.
 # These parameters might require optimization for your specific type of dataset!
-from gaitmap.stride_segmentation.hmm import SimpleHmm
-
-stride_model = SimpleHmm(
-    n_states=20,
-    n_gmm_components=6,
-    algo_train="baum-welch",
-    stop_threshold=1e-9,
-    max_iterations=5,
-    architecture="left-right-strict",
-    verbose=True,
-    name="stride_model",
-)
-
-transition_model = SimpleHmm(
-    n_states=5,
-    n_gmm_components=3,
-    algo_train="baum-welch",
-    stop_threshold=1e-9,
-    max_iterations=5,
-    architecture="left-right-loose",
-    verbose=True,
-    name="transition_model",
+from gaitmap.stride_segmentation.hmm import CompositeHmmConfig, HmmSubModelConfig, RothHmmConfig
+
+model_config = CompositeHmmConfig(
+    modules=(
+        HmmSubModelConfig(
+            name="transition",
+            role="transition",
+            n_states=5,
+            n_gmm_components=3,
+            algo_train="baum-welch",
+            stop_threshold=1e-9,
+            max_iterations=5,
+            architecture="left-right-loose",
+            verbose=True,
+        ),
+        HmmSubModelConfig(
+            name="stride",
+            role="stride",
+            n_states=20,
+            n_gmm_components=6,
+            algo_train="baum-welch",
+            stop_threshold=1e-9,
+            max_iterations=5,
+            architecture="left-right-strict",
+            verbose=True,
+        ),
+    )
 )
 
 # %%
@@ -116,19 +121,19 @@
 # invoke the training process.
 # Again, all configurable parameters are exposed for demonstration purpose.
 # These parameters should again work for most usecases.
-from gaitmap.stride_segmentation.hmm import RothSegmentationHmm
+from gaitmap.stride_segmentation.hmm import PreTrainedRothSegmentationModel, RothSegmentationHmm
 
 segmentation_model = RothSegmentationHmm(
-    stride_model=stride_model,
-    transition_model=transition_model,
-    feature_transform=feature_transform,
-    algo_predict="viterbi",
-    algo_train="baum-welch",
-    stop_threshold=1e-9,
-    max_iterations=1,
-    initialization="labels",
-    verbose=True,
-    name="segmentation_model",
+    hmm_config=RothHmmConfig(
+        model_config=model_config,
+        feature_transform=feature_transform,
+        algo_predict="viterbi",
+        algo_train="baum-welch",
+        stop_threshold=1e-9,
+        max_iterations=1,
+        initialization="labels",
+        name="segmentation_model",
+    ),
 )
 
 # %%
@@ -139,12 +144,18 @@
 # convention!).
 # The main input format for the training process are gait sequences which include transitions as well as valid strides.
 # To train on multiple sequences, we can just feed a list of gaitsequences into the model for training.
-# For each gait sequence we also need to have a valid stride list. In this example we handle the data from the left and
-# right foot as separate gait sequences and add them to a simple list.
-# We have to do the same for the stride lists.
+# For each gait sequence we also need typed training regions with `start`, `end`, and `type`.
+# In this example the stride regions are all of type `"stride"` and transitions are defined implicitly as everything
+# not covered by a region.
+# We handle the data from the left and right foot as separate gait sequences and add them to a simple list.
 
 data_train_sequence = [bf_data["left_sensor"], bf_data["right_sensor"]]
-stride_list_sequence = [stride_list["left_sensor"], stride_list["right_sensor"]]
+region_list_sequence = []
+for sensor in ["left_sensor", "right_sensor"]:
+    region_list = stride_list[sensor][["start", "end"]].copy()
+    region_list.insert(0, "roi_id", np.arange(len(region_list)))
+    region_list["type"] = "stride"
+    region_list_sequence.append(region_list.set_index("roi_id"))
 
 # %%
 # Training
@@ -156,24 +167,34 @@
 # The model will internally perform the feature transformation of the dataset, train the individual sub models and
 # finally combine them to a flatted segmentation model.
 
-segmentation_model = segmentation_model.self_optimize(
-    data_train_sequence, stride_list_sequence, sampling_rate_hz=sampling_rate_hz
-)
+if segmentation_model.backend.backend_id == "scipy-inference":
+    print(
+        "Skipping HMM training because the current environment only provides the "
+        "SciPy inference backend. Falling back to the packaged pre-trained model."
+    )
+    segmentation_model = PreTrainedRothSegmentationModel()
+else:
+    segmentation_model = segmentation_model.self_optimize(
+        data_train_sequence, region_list_sequence, sampling_rate_hz=sampling_rate_hz
+    )
 
 # %%
 # Inspecting the Results
 # --------------------------------------
 #
-# Now all internal models which were initialized as "None" should be populated by pomegranate models.
-# We can now have a look at the final transition matrix or the trained distributions (GMMs).
+# Now the trained model is stored as a serializable HMM state.
+# We can now have a look at the final transition matrix, the backend provenance, or one of the trained emission
+# distributions.
 # You could now either use the model to predict stride borders on an unseen sequence or save it to a json file for later
 # use.
 
 np.set_printoptions(precision=3, linewidth=180, suppress=True)
 
-print(segmentation_model.model.dense_transition_matrix()[0:-2, 0:-2])
+print(segmentation_model.model.trained_with)
 
-print(segmentation_model.model.states[10])
+print(segmentation_model.model.compiled.graph.transition_probs)
+
+print(segmentation_model.model.compiled.emissions[10])
 
 # %%
 # Applying the Model to a Sequence
@@ -183,18 +204,22 @@
 # We will also plot the results to see how well the model performs.
 from gaitmap.stride_segmentation.hmm import HmmStrideSegmentation
 
-hmm = HmmStrideSegmentation(segmentation_model).segment(bf_data, sampling_rate_hz=sampling_rate_hz)
+# Note: We are using a high snap_to_min_win_ms here to get consistent output agaisnt all HMM backends.
+#       They have slight inconsitencies in some edge cases and we want to make sure this example provides the same results for consistent snapshots.
+hmm = HmmStrideSegmentation(segmentation_model, snap_to_min_win_ms=300).segment(
+    bf_data, sampling_rate_hz=sampling_rate_hz
+)
 hmm.stride_list_
 
 # %%
 # Plotting the Results
 # --------------------
-sensor = "left_sensor"
+sensor = "right_sensor"
 
 fig, axs = plt.subplots(nrows=2, sharex=True, figsize=(10, 5))
 axs[0].set_title("gaitmap Body Frame Dataset")
 axs[0].plot(bf_data.reset_index(drop=True)[sensor]["gyr_ml"])
-for start, end in hmm.stride_list_["left_sensor"].to_numpy():
+for start, end in hmm.stride_list_[sensor].to_numpy():
     axs[0].axvline(start, c="r")
     axs[0].axvline(end, c="r")
     axs[0].axvspan(start, end, alpha=0.2)

examples/trajectory_reconstruction/trajectory_reconstruction_region.py

@@ -28,7 +28,7 @@
 
 imu_data = get_healthy_example_imu_data()
 dummy_regions_list = pd.DataFrame([[0, len(imu_data["left_sensor"])]], columns=["start", "end"]).rename_axis("gs_id")
-dummy_regions_list = {k: dummy_regions_list for k in get_multi_sensor_names(imu_data)}
+dummy_regions_list = dict.fromkeys(get_multi_sensor_names(imu_data), dummy_regions_list)
 dummy_regions_list["left_sensor"]
 
 

examples/trajectory_reconstruction/zupt_dependency.py

@@ -21,7 +21,6 @@
 
 import pandas as pd
 
-
 # %%
 # The Data
 # --------

notes/hmm_refactor_plan_overview.md

@@ -0,0 +1,17 @@
+# HMM Refactor Plan Overview
+
+This note replaces the earlier single investigation note with a staged plan.
+
+The refactor should happen in three steps:
+
+1. Move the current HMM system to a unified input config while still storing and operating on `pomegranate` models.
+2. Introduce a generic serializable `HMMState` and change `RothSegmentationHmm.model` to use that state.
+3. Add a dedicated `pomegranate 0.14` backend abstraction that converts between backend-native models and `HMMState`.
+
+Why this order:
+
+- Step 1 changes the public construction/configuration surface without changing the trained-model representation.
+- Step 2 changes the trained-model representation while still keeping the current implementation behavior.
+- Step 3 isolates the legacy backend after the public config and model-state surfaces are already stable.
+
+This order reduces risk and keeps regressions easier to localize.

notes/hmm_refactor_step1_config.md

@@ -0,0 +1,84 @@
+# Step 1: Unified Config Surface
+
+## Goal
+
+Replace the dedicated `stride_model` and `transition_model` init parameters with a single composite model config while
+still using the current `pomegranate` model objects internally.
+
+## Target API
+
+The HMM class should accept one nested config object that describes:
+
+- the available submodules,
+- each submodule's local HMM settings,
+- how submodules are connected,
+- and how the combined model is trained.
+
+Example target surface:
+
+```python
+class HmmSubModelConfig(_BaseSerializable):
+    n_states: int
+    n_gmm_components: int
+    architecture: Literal["left-right-strict", "left-right-loose", "fully-connected"]
+    algo_train: Literal["viterbi", "baum-welch", "labeled"]
+    stop_threshold: float
+    max_iterations: int
+    name: str
+
+
+class HmmConnectionConfig(_BaseSerializable):
+    from_module: str
+    from_state: int
+    to_module: str
+    to_state: int
+    initial_probability: float | None = None
+
+
+class CompositeHmmConfig(_BaseSerializable):
+    modules: dict[str, HmmSubModelConfig]
+    connections: tuple[HmmConnectionConfig, ...]
+    initialization: Literal["labels", "fully-connected"]
+    combined_algo_train: Literal["viterbi", "baum-welch", "labeled"]
+    combined_stop_threshold: float
+    combined_max_iterations: int
+    name: str
+```
+
+## Scope
+
+This step should only change configuration flow.
+It should not yet change:
+
+- `RothSegmentationHmm.model`
+- the stored pretrained model format
+- the use of raw `pomegranate` models internally
+- `_HackyClonableHMMFix`
+- `gaitmap/base.py` HMM serialization
+
+## Implementation Notes
+
+- Keep the current Roth behavior by providing a default two-module config with `transition` and `stride`.
+- Move the current submodel-specific parameters into default config values.
+- Keep the current training loop structure:
+  - train each configured submodule independently
+  - combine them by building a final `pomegranate` model
+  - run the final refinement pass on the combined model
+- The training/data-splitting logic can still be Roth-specific in this step.
+  The point is to stabilize the input surface first.
+
+## Compatibility Strategy
+
+- Backwards compatibility for init parameters is optional.
+- If needed, provide a thin compatibility shim that maps legacy `stride_model` / `transition_model` inputs to the new
+  config internally.
+- The stored model format should remain unchanged in this step so that the pretrained artifact continues to load
+  without migration work.
+
+## Expected Outcome
+
+After this step:
+
+- the public constructor uses a single config object,
+- custom multi-module setups become expressible,
+- but the trained model is still a `pomegranate` model and behavior should be unchanged.