new cluster test API

[CarinaFo]

set up new cluster_test api that sets up design matrix based on Wilkinson formula (using formulaic package)
Update (01/26):

• PR is currently not actively worked on due to time constraints

[drammock]

just a few comments to get you rolling. Maybe ping us when the formulaic bit is added? (or sooner if you have any questions in the meantime, of course!)

[hoechenberger]

This might be related:

mne-tools/mne-incubator#31

cc @SophieHerbst

[cbrnr]

I think it is a great idea to revamp the current cluster permutation test API 🚀! Could you please share the reasoning behind choosing a pandas DataFrame as the container for evokeds and related meta info? If possible, I'd try to avoid using pandas (an optional dependency) when something similar could be achieved using, for example, a simple dictionary.

[CarinaFo]

@cbrnr I think the main reason for a dataframe instead of a dictionary is that formulaic only allows for dataframes as input and we want to include Wilkinson formula support in the new cluster_test API.

[hoechenberger]

@cbrnr I think the main reason for a dataframe instead of a dictionary is that formulaic only allows for dataframes as input and we want to include Wilkinson formula support in the new cluster_test API.

But this doesn't need to be user-facing, then, no? Just trying to understand. If a user passes in a list of TypedDicts or Dataclasses, you can internally create the DataFrames that need to be passed to formulaic. The user would then also get tab-completion assistance in their editor. But it's just a thought. Great work so far in any case!

[drammock]

you can internally create the DataFrames that need to be passed to formulaic

This would still require pandas to be available though.

[hoechenberger]

you can internally create the DataFrames that need to be passed to formulaic

This would still require pandas to be available though.

Sure
But it would provide a potentially more user-friendly API

[drammock]

you can internally create the DataFrames that need to be passed to formulaic

This would still require pandas to be available though.

Sure But it would provide a potentially more user-friendly API

For context, this is step 1 of a GSoC project. A later step involves (probably) creating helper functions that will create the necessary DataFrame for the user. That's not done here because:

1. there are probably complicated use cases / designs that we won't foresee, so we want it to be possible for the user to pass in their own custom dataframe instead of our internally-generated one.
2. we're actually not sure how helpful the helper function will be (at least in some cases): if what you need to pass in is a set of matched lists of subject IDs, evoked objects, and condition names, well then you might as well just call DataFrame(dict(subj=subj_list, cond=cond_list, data=evk_list)) yourself instead of calling the helper function. A helper function makes much more sense in other cases, like when dealing with Epochs objects where the conditions are intermixed within one object.

[larsoner]

@CarinaFo I pushed a commit to add the dataset and add formulaic to our full and doc dependencies so that eventually CIs can use them properly. I added the tags [skip azp] [skip actions] to skip running those CIs. Feel free to git pull the changes back to your local machine!

If you look at the CI runs, you can see that CircleCI, which builds modified examples/tutorials in PRs, hit an error up on 9c8ec90:

sphinx.errors.ExtensionError: Could not find docstring in file "/home/circleci/project/tutorials/stats-sensor-space/76_new_cluster_test_api.py". A docstring is required by sphinx-gallery unless the file is ignored by "ignore_pattern"

Then I pushed a little commit to fix that in 47363b5, and now it hits a different error (which replicates what I saw locally when I tried to run the example):

../tutorials/stats-sensor-space/76_new_cluster_test_api.py unexpectedly failed to execute correctly:

    Traceback (most recent call last):
      File "/home/circleci/project/tutorials/stats-sensor-space/76_new_cluster_test_api.py", line 449, in <module>
        df_long = convert_wide_to_long(df)
      File "/home/circleci/project/tutorials/stats-sensor-space/76_new_cluster_test_api.py", line 431, in convert_wide_to_long
        data_2d = row["data"]
      File "/home/circleci/python_env/lib/python3.10/site-packages/pandas/core/series.py", line 1121, in __getitem__
        return self._get_value(key)
      File "/home/circleci/python_env/lib/python3.10/site-packages/pandas/core/series.py", line 1237, in _get_value
        loc = self.index.get_loc(label)
      File "/home/circleci/python_env/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
        raise KeyError(key) from err
    KeyError: 'data'

It would be good to keep CircleCI green at least so we can see renderings of the tutorial with each push. If you make sure python -i tutorials/stats-sensor-space/76_new_cluster_test_api.py runs cleanly locally each time before you push then it should work on CircleCI now as well!

[scott-huberty]

@drammock in 5756853 @larsoner tests the results of our new cluster_test (and thus really mne.stats.ttest_1samp_no_p, mne.stats.f_oneway mne.stats.f_mway_rm) against equivalent Fieldtrip functions.

In mne/stats/tests/test_cluster_level_modern.py we test the results of cluster_test against the existing mne.stats.ttest_1samp_no_p, mne.stats.f_oneway mne.stats.f_mway_rm (which cluster_test calls under the hood anyways..).

Is this convincing enough to go ahead and update tutorials/stats-sensor-space/{40, 50, 70, 75}_*.py to use the cluster_test API instead of the now legacy functions? Or do you want to wait until a later date.

[CarinaFo]

Great work @scott-huberty and awesome to have the test against the fieldtrip implementation @larsoner. I will review the changes and can definitely contribute to the new tutorial.

[drammock]

s this convincing enough to go ahead and update tutorials/stats-sensor-space/{40, 50, 70, 75}_*.py to use the cluster_test API instead of the now legacy functions?

yes go for it

[scott-huberty]

Per a discussion with @wmvanvliet and @CarinaFo - we agreed to strip out the plotting methods of ClusterResult and will propose them in a subsequent PR.

[scott-huberty]

@wmvanvliet do you want to git pull and try cluster_test with your data gain?

[wmvanvliet]

works like a charm!

[wmvanvliet]

results.stat_obs still has shape n_times x n_channels, and it would be nice if that were n_channels x n_times

[larsoner]

Sounds like we need some unit tests that our output shapes match our input shapes...

[scott-huberty]

results.stat_obs still has shape n_times x n_channels, and it would be nice if that were n_channels x n_times

If I am not mistaken That probably means we'll have to swap results.clusters as well e.g.

(array([0]), array([3])) -> (array([3]), array([0]))

[larsoner]

One thing we should remember to check... the docs don't say adjacency can be None, but that is its default -- maybe that means "auto" and if Evokeds are passed uses channels, if ndarray is passed assumes lattice? Another option would be to make the default "auto" which means "infer from container if possible, and error otherwise". Not 100% sure what the safest/best option is here. We get to pick and not worry about backward compat so we should try to get it right...

[scott-huberty]

One thing we should remember to check... the docs don't say adjacency can be None, but that is its default -- maybe that means "auto" and if Evokeds are passed uses channels, if ndarray is passed assumes lattice? Another option would be to make the default "auto" which means "infer from container if possible, and error otherwise". Not 100% sure what the safest/best option is here. We get to pick and not worry about backward compat so we should try to get it right...

I checked and I think the adjacency docstring is shared with our legacy functions via the doc dict, which accounts for the misinformation. Probably worth making its docstring stand alone.

I think “auto” sounds nice and is very explicit.

+1 to merge after the next release of MNE so that we get 3 months to play around with it and make changes (if needed) without deprecation cycles,