[FEAT] MEDIC distortion correction via warpkit - #541
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #541 +/- ##
==========================================
+ Coverage 84.17% 84.21% +0.04%
==========================================
Files 30 33 +3
Lines 2938 3104 +166
Branches 391 326 -65
==========================================
+ Hits 2473 2614 +141
- Misses 388 414 +26
+ Partials 77 76 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
57ccc47 to
21ab1c6
Compare
bcaf4ad to
9ecfbae
Compare
Three additions targeting the codecov/patch failure on nipreps#541: * sdcflows/interfaces/tests/test_warpkit.py (new) — instantiates every warpkit-backed interface so its spec class body is hit at import (was 0% on codecov despite running locally; likely an xdist worker-merge artefact). Also covers ``_as_str_list``, the ``_pkg`` invariant, and the ``border_filt=(1, 5)`` traits default regression. * sdcflows/workflows/tests/test_outputs.py (new) — direct construction test for ``init_fmap_derivatives_wf``, exercising both the default and ``write_dynamic=True`` paths. The MEDIC dynamic sinks branch was only reached transitively from the ``test_fmap_wf`` slow test, hence 0% patch coverage on outputs.py. * sdcflows/workflows/fit/tests/test_medic.py — fill the remaining helper gaps: empty-metadata rejection, ``sloppy=True`` zooms_min, ``_first``, and ``_temporal_mean`` for both 3D and 4D inputs. The ``_run_interface`` method bodies in interfaces/warpkit.py remain uncovered in the fast/slow envs since they require warpkit to import; the existing ``veryslow`` MEDIC fixtures exercise them when the optional dependency is installed.
|
I have no idea why test (3.13, pre, fast) is failing on the cache step. It looks like an out-of-space error on the runner (but the other jobs with the same steps succeeded, so very confused): |
|
@vanandrew I think someone must have rerun that failing job and now it's passing. |
tsalo
left a comment
There was a problem hiding this comment.
Thanks for this! I made a quick pass through the code.
|
Pushed four commits addressing your feedback (noise_frames, ctor inputs, nitransforms switch + interface cleanup, unified fmap output). |
|
I can't tell if these test failures are something wrong with my dataset additions or if it's just another one-off error: |
a4c4781 to
b070878
Compare
|
Looks like the CI keeps failing because it's running out of space (I added two datasets for tests so that might have pushed it over the edge). I pushed 6f5cfff2 adding a - name: Free disk space
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: false
swap-storage: falseThis gets rid of some unneeded development tools on the github runner to free room for data. |
|
@tsalo Alright. I made changes based on our convo today:
|
Three additions targeting the codecov/patch failure on nipreps#541: * sdcflows/interfaces/tests/test_warpkit.py (new) — instantiates every warpkit-backed interface so its spec class body is hit at import (was 0% on codecov despite running locally; likely an xdist worker-merge artefact). Also covers ``_as_str_list``, the ``_pkg`` invariant, and the ``border_filt=(1, 5)`` traits default regression. * sdcflows/workflows/tests/test_outputs.py (new) — direct construction test for ``init_fmap_derivatives_wf``, exercising both the default and ``write_dynamic=True`` paths. The MEDIC dynamic sinks branch was only reached transitively from the ``test_fmap_wf`` slow test, hence 0% patch coverage on outputs.py. * sdcflows/workflows/fit/tests/test_medic.py — fill the remaining helper gaps: empty-metadata rejection, ``sloppy=True`` zooms_min, ``_first``, and ``_temporal_mean`` for both 3D and 4D inputs. The ``_run_interface`` method bodies in interfaces/warpkit.py remain uncovered in the fast/slow envs since they require warpkit to import; the existing ``veryslow`` MEDIC fixtures exercise them when the optional dependency is installed.
da9d07b to
c55efd0
Compare
Three additions targeting the codecov/patch failure on nipreps#541: * sdcflows/interfaces/tests/test_warpkit.py (new) — instantiates every warpkit-backed interface so its spec class body is hit at import (was 0% on codecov despite running locally; likely an xdist worker-merge artefact). Also covers ``_as_str_list``, the ``_pkg`` invariant, and the ``border_filt=(1, 5)`` traits default regression. * sdcflows/workflows/tests/test_outputs.py (new) — direct construction test for ``init_fmap_derivatives_wf``, exercising both the default and ``write_dynamic=True`` paths. The MEDIC dynamic sinks branch was only reached transitively from the ``test_fmap_wf`` slow test, hence 0% patch coverage on outputs.py. * sdcflows/workflows/fit/tests/test_medic.py — fill the remaining helper gaps: empty-metadata rejection, ``sloppy=True`` zooms_min, ``_first``, and ``_temporal_mean`` for both 3D and 4D inputs. The ``_run_interface`` method bodies in interfaces/warpkit.py remain uncovered in the fast/slow envs since they require warpkit to import; the existing ``veryslow`` MEDIC fixtures exercise them when the optional dependency is installed.
c55efd0 to
b0a6904
Compare
|
@tsalo I did a rebase to the latest main, hopefully to quash the 3.13 test error that happened in the last CI run: https://github.com/nipreps/sdcflows/actions/runs/26792754492/job/79155816998#step:22:320 |
Three additions targeting the codecov/patch failure on nipreps#541: * sdcflows/interfaces/tests/test_warpkit.py (new) — instantiates every warpkit-backed interface so its spec class body is hit at import (was 0% on codecov despite running locally; likely an xdist worker-merge artefact). Also covers ``_as_str_list``, the ``_pkg`` invariant, and the ``border_filt=(1, 5)`` traits default regression. * sdcflows/workflows/tests/test_outputs.py (new) — direct construction test for ``init_fmap_derivatives_wf``, exercising both the default and ``write_dynamic=True`` paths. The MEDIC dynamic sinks branch was only reached transitively from the ``test_fmap_wf`` slow test, hence 0% patch coverage on outputs.py. * sdcflows/workflows/fit/tests/test_medic.py — fill the remaining helper gaps: empty-metadata rejection, ``sloppy=True`` zooms_min, ``_first``, and ``_temporal_mean`` for both 3D and 4D inputs. The ``_run_interface`` method bodies in interfaces/warpkit.py remain uncovered in the fast/slow envs since they require warpkit to import; the existing ``veryslow`` MEDIC fixtures exercise them when the optional dependency is installed.
BIDS keeps trailing noise frames in separate noRF files, so SDCFlows shouldn't carry an interface input for them — let warpkit's default of zero apply.
Passing n_cpus and debug through the interface constructor keeps node construction in one place and removes the post-init mutation pattern.
…sforms The dynamic-fieldmap apply workflow was the only consumer of ``ApplyWarp``/``ConvertFieldmap`` in warpkit; rewrite it on top of ``sdcflows.transform.apply_dynamic_unwarp``, a per-frame extension of the existing scipy/nitransforms-backed resampling that powers the static path. With that internal consumer gone, the warpkit interface module shrinks to just the two stages SDCFlows actually runs (``UnwrapPhase``, ``ComputeFieldmap``); the unused ``MEDIC``, ``ApplyWarp``, ``ConvertWarp``, ``ConvertFieldmap`` and ``ComputeJacobian`` wrappers, along with the helpers that only served them, are removed.
``init_medic_wf`` previously emitted both a 3D static ``fmap`` (a B-spline-smoothed temporal mean of the dynamic field) and a separate 4D ``fmap_dynamic``, alongside parallel ``fmap_dynamic_ref`` and ``fmap_dynamic_mask`` plumbing in the base preproc workflow and a ``write_dynamic`` branch in the derivatives sinks. Per review, collapse all of that to a single ``fmap`` output whose dimensionality (3D for static estimators, 4D for MEDIC) tells consumers which apply path to use. The B-spline shim is gone (MEDIC's field is already on the EPI grid), the derivatives sink uses ``MergeSeries(allow_4D=True)`` to pass 3D and 4D fmaps through the same path, and the per-volume apply workflow renames its input from ``fmap_dynamic`` to ``fmap`` to match. Downstream tools that don't yet handle 4D field maps must block MEDIC-based estimators until they do.
The static init_magnitude_wf averaged across time before brain-extracting, which collapsed MEDIC's per-volume magnitude into one 3D ref/mask. The new init_dynamic_magnitude_wf splits the 4D first-echo magnitude, runs the IntensityClip/N4/BrainExtraction chain per frame, and re-concatenates so fmap_ref and fmap_mask are 4D and track the per-volume fieldmap.
…nt accepted kwargs Adds three regression tests around the MEDIC guard clauses in ``FieldmapEstimation.__attrs_post_init__`` (single-part input, single-echo pair, mismatched mag/phase counts), and documents the two ``init_medic_wf`` parameters (``use_metadata_estimates``, ``fallback_total_readout_time``) that are accepted via ``init_fmap_preproc_wf`` for parity but currently ignored.
…ation.is_dynamic Replaces the ``estimator.method == EstimatorType.MEDIC`` check in ``init_fmap_preproc_wf`` with a new ``FieldmapEstimation.is_dynamic`` property backed by a module-level ``_DYNAMIC_METHODS`` set in ``sdcflows.fieldmaps``. Future per-volume estimators (DOCMA, TOAST, ...) only need to be added to ``_DYNAMIC_METHODS`` — the shared workflow no longer carries per-method branching.
Adds ``test_wrangler_medic_trigger`` covering the three modes of MEDIC discovery against synthetic BIDS skeletons in a single parametrized test: 1. ``default-IntendedFor`` — sidecars carry ``IntendedFor`` and the default discovery path picks them up. 2. ``force_medic`` — sidecars carry no metadata; the explicit flag short-circuits the default ``IntendedFor`` gate. 3. ``baseline-no-trigger`` — no metadata, no override, no fmapless; MEDIC must refuse to fire so runs without expected metadata are not silently picked up. The existing ``test_wrangler_force_medic_without_intended_for`` covers cases 2 and 3 implicitly via ``medic_no_intended_for``; this new test pins all three side-by-side and was validated against a real ds006926/sub-a01 layout before being formalized here.
The bumped data-cache-v3 (now includes ds006926 + ds007637) tips the 3.10/min/fast lane over the 14 GB root-disk limit during cache restore. Free ~20 GB up front by purging the Android SDK, .NET, and Haskell tool-caches; keep the tool-cache, swap, and apt large-packages alone so later steps (apt-cache restore, conda, uv) aren't disturbed.
…arpkit masks The MEDIC workflow now exposes the first-echo magnitude series untouched as ``fmap_ref`` (the ``pick_mag1`` output) and routes warpkit's per-frame ``UnwrapPhase`` masks straight to ``fmap_mask``, instead of running a per-frame N4 + skull-strip pass to synthesize both. ``init_dynamic_magnitude_wf`` was only used by MEDIC, so it is removed from ``fit/fieldmap.py`` entirely. The expensive per-volume MapNode work (N4, intensity clip, brain extraction over every frame) is gone with it. Updates ``test_medic_construct`` to drop the removed ``magnitude_wf`` node.
…DIC-first ordering Make MEDIC discovery follow the BIDS fieldmap-intent model rather than file structure: * MEDIC is now discovered only from declared intent metadata -- a complex multi-echo BOLD is picked up via its ``B0FieldIdentifier`` (the self-referential pattern BIDS endorses for images that estimate their own B0 field, as in pepolar) through the existing Step 1 path, or via legacy ``IntendedFor`` in the dedicated MEDIC block. The structure-only auto-discovery (and the ``force_medic`` flag that enabled it) is removed: part-mag/part-phase alone no longer triggers MEDIC. * Add ``no_medic`` to disable MEDIC discovery via either route (kwarg + ``--no-medic`` CLI flag + ``config.workflow.no_medic``); it also skips a MEDIC-shaped ``B0FieldIdentifier`` group in Step 1. Return estimators in a deterministic order: ``estimators.sort(key=lambda e: (not e.is_dynamic, e.bids_id))``. Step 1 iterates a ``set`` of ``B0FieldIdentifier``s, so the prior order was hash-seed dependent. The sort also encodes one intentional, documented priority -- dynamic (MEDIC) estimators come first, so a consumer selecting the first applicable estimator per target prefers MEDIC over a coexisting static fieldmap. Fieldmap-less ANAT estimators are appended afterwards and stay last. Tests: cover the B0FieldIdentifier and IntendedFor MEDIC routes, a guard that structure alone does not fire MEDIC, and that MEDIC sorts ahead of a coexisting PEPOLAR estimator.
Address review: replace the fmt: off/on pair around workflow.connect with an inline # fmt:skip, and use pathlib.Path over os.path for the corrected output. Also normalize the warpkit module/test copyright headers to the unversioned NiPreps form.
Address review: collapse the unused use_metadata_estimates and fallback_total_readout_time parity args into **kwargs (dropping the now-dead del statement and their docstring entries), and inline the single-use _MEDIC_DESC string at the workflow.__desc__ assignment.
Address review: move the duplicated _MEDIC_TEST_VOLUMES, _truncate_to_volumes helper, and MEDIC_FIXTURES list out of the fit/apply test modules into shared conftest fixtures (medic_test_volumes, truncate_to_volumes, and a parametrized medic_fixture). Both end-to-end tests now consume the fixtures, removing the keep-in-sync duplication. Also normalize remaining copyright headers.
Move warpkit into the core dependencies (keeping the python_version >= '3.11' marker so 3.10 installs still resolve) rather than gating it behind an opt-in extra. Drop the [warpkit] optional-dependencies group and the now-redundant veryslow: warpkit tox extra. The non-commercial WUSTL license is documented in the init_medic_wf module docstring.
Address review: the helper recomputed fmap_hz * ro_time internally, duplicating the VSM already computed in _sdc_unwarp. Change the signature to accept the VSM and reuse it at the call site, keeping the 3D/4D-capable helper for downstream use.
nipype prunes a node's working directory to the files referenced by its string-valued outputs. _dynamic_unwarp returned a PosixPath, which the pruning did not recognize, so corrected.nii.gz was deleted before the downstream average node could read it. Return str() of the path.
Collapse the parallel dynamic apply stack into the static machinery: - unwarp_parallel accepts a 3D or 4D field; a 3D (shared) field is np.broadcast_to-viewed across frames so per-frame selection is a single branchless fmap_hz[..., volid]. The 3D case is the degenerate 4D case. - B0FieldTransform can be constructed from a pre-gridded field (mapped=) in addition to B-spline coeffs; apply() dispatches on provenance (coeffs -> fit(); pre-gridded -> use as-is), then both routes share one _resample_with_fieldmap helper (the formerly duplicated tail). Guard added for the empty (no coeffs, no mapped) case. - Drop _dynamic_unwarp_parallel and the apply_dynamic_unwarp wrapper; the MEDIC apply node constructs B0FieldTransform(mapped=...).apply(...) directly, mirroring how ApplyCoeffsField drives the static path with coeffs.
Address review feedback on the warpkit-backed interfaces: - Rename the `n_cpus` trait to `num_threads` (default 1, nohash) on both UnwrapPhase and ComputeFieldmap so nipype syncs it with the Node's `n_procs`; drop the explicit `n_cpus=omp_nthreads` in the MEDIC workflow. - Drop the `list(...)` casts (InputMultiObject already yields a list) and replace `isdefined(...) else None` with `... or None`. - Remove the ValueError->RuntimeError try/except; let errors propagate. - Add a `desc` to `svd_filt`.
Separate the two concepts the old docstring conflated: fmap_hz must be
co-gridded with moving (same voxel lattice, so the per-voxel scaling and
coordinate shift broadcast element-wise) AND valued in the undistorted
(target) sense. The prior wording ("already on the moving grid") read as
source/distorted space, which it is not.
warpkit 1.5.0 removes the ``wrap_limit`` argument from
``warpkit.api.unwrap_phase``, so forwarding it raised
TypeError: unwrap_phase() got an unexpected keyword argument 'wrap_limit'
inside the ``unwrap`` node, failing test_dynamic_unwarp_run on both
fixtures.
The flag only existed to disable a cascade of hardcoded field-magnitude
thresholds, which warpkit replaced with an automatic 2-pi branch
selector (vanandrew/warpkit#32). There is no successor input to thread
through, so the trait is dropped rather than renamed. Nothing in
SDCFlows ever set it.
71f7ba7 to
aab352e
Compare
Summary
Adds multi-echo dynamic distortion correction (MEDIC) to sdcflows, backed by warpkit. MEDIC estimates a per-volume B0 fieldmap directly from a multi-echo, mag+phase BOLD series, capturing breathing- and motion-driven field changes that a single static fieldmap can't.
Revives #435 / #438 with the simpler pure-Python warpkit (post vanandrew/warpkit#16, no Julia/C++ setup required). Closes #36.
What's new
Workflows
init_medic_wf(sdcflows/workflows/fit/medic.py) — multi-echo phase + magnitude → 4D Hz fieldmap (one volume per timepoint, on the EPI grid) + brain-extracted reference + brain mask. Two-stage warpkit call (UnwrapPhase→ComputeFieldmap) so the per-frame masks stay accessible. Singlefmapoutput is 4D for MEDIC, leaving the 3D-vs-4D dispatch to the apply consumer.init_dynamic_unwarp_wf(sdcflows/workflows/apply/dynamic.py) — per-volume apply path built onsdcflows.transform.apply_dynamic_unwarp, a per-frame extension of the same scipy/nitransforms-backed resampling that powers the staticinit_unwarp_wf. No warpkit needed at runtime. Includes Jacobian determinant intensity correction (jacobian=Truedefault) sharingtransform.fieldmap_jacobianwith the static path.Interfaces
sdcflows/interfaces/warpkit.py— thinLibraryBaseInterfacewrappers around the two MEDIC stages SDCFlows actually drives:UnwrapPhase(ROMEO) andComputeFieldmap. Lazy import — sdcflows imports warpkit only when these interfaces actually run.Detection / dispatch
EstimatorType.MEDICadded.FieldmapEstimation.__attrs_post_init__detects MEDIC inputs (part-{phase,mag}onbold/epi/sbrefsources), enforces matched echo cardinality (≥2 phase, equal mag count), and rejects partial or mixed part sets.get_workflowinstantiatesinit_medic_wfwithEchoTime-sorted lists (BIDS doesn't guaranteeechoentity == numeric order).part='phase'andpart='mag'queries (some datasets carryIntendedForonly on one side); dedup walks the full sibling set, no reliance on pybids ordering.force_medicopt-in onfind_estimators— auto-discover MEDIC estimators from complex multi-echo BOLD even when neitherIntendedFornorB0FieldIdentifieris set. Pairing is unambiguous because the part-mag/part-phase echoes of the same run are MEDIC sources by construction. Intended for public datasets that ship the required echoes without the metadata the default discovery path needs.Plumbing
init_fmap_preproc_wfskipsfmap_coefffor MEDIC (the fieldmap is on the EPI grid by construction, no B-spline rep).init_fmap_derivatives_wfusesMergeSeries(allow_4D=True)so MEDIC's 4Dfmappasses through the sameds_fieldmapsink as the 3D static fmaps.Packaging
warpkitextra inpyproject.toml, explicitly excluded from[all]because warpkit ships under a non-commercial WUSTL license. Defaultpip install sdcflowsstays Apache-clean.tox.ini: theveryslowenv pulls the warpkit extra so MEDIC end-to-end tests only run there.v3).Validation
echo=Query.REQUIRED),FieldmapEstimationcardinality check (len(phase_files) < 2), and_unpack_metadataruntime guard insideinit_medic_wf.test_apply_dynamic_unwarp_matches_staticpinsapply_dynamic_unwarpto the same Hz→VSM + scipy.ndimage convention as the static_sdc_unwarppath — catches drift in sign / pe_info handling.test_wrangler_filter/test_wrangler_URIsparametrized with a 3-session × 3-echo × {mag,phase} BIDS skeleton.Known compromises
IntendedFor/B0FieldIdentifier— same constraint as the existing single-PE EPI branch. Datasets missing both can opt into discovery viaforce_medic=Trueonfind_estimators(see Detection / dispatch above).fmapoutput is 4D for MEDIC. Downstream tools that expect 3D field maps need to either dispatch on dimensionality or block MEDIC-based estimators until they do.Test plan
pytest sdcflows/utils/tests/test_wrangler.py— wrangler MEDIC detection paths (includingforce_medic)pytest sdcflows/workflows/fit/tests/test_medic.py—init_medic_wfconstruction +_unpack_metadataguardspytest sdcflows/workflows/apply/tests/test_dynamic.py—init_dynamic_unwarp_wfconstruction + jacobian flag + per-frame resampling vs. static pathpytest -m veryslowwithpip install sdcflows[warpkit]and ds006926 / ds007637 fixtures present — full MEDIC fit + dynamic apply