Skip to content

perf(float32): make the data path float32 at its source - #344

Merged
daharoni merged 2 commits into
masterfrom
perf/float32-data-path
Jun 26, 2026
Merged

perf(float32): make the data path float32 at its source#344
daharoni merged 2 commits into
masterfrom
perf/float32-data-path

Conversation

@daharoni

@daharoni daharoni commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

What

Phase 1 of moving minian off float64: switch the data path to float32 at its source. Because dask/xarray/numpy infer output dtype from their inputs, flipping the source is enough for the bulk of the pipeline to follow.

Two changes:

  1. io.load_videos default np.float64 -> np.float32. The footage is 8-bit, so every loaded value is represented exactly in float32 (no precision loss at load), while halving memory and on-disk size. Callers that pass dtype explicitly - including the demo notebook (uint8) and every test - are unaffected.
  2. Pipeline notebook: Y.astype(float) -> Y.astype(np.float32) at the Y_fm_chk save. This was the single spot promoting the demo movie to float64 - the ~69 GB-vs-34 GB cube behind the slow save/rechunk in feat(pipeline-nb): expose rechunker mem_limit as a tunable parameter #342. Everything downstream (Y_fm_chk, Y_hw_chk, seed init, CNMF prep) is now float32; all other notebook casts were already float32.

Scope (deliberately limited)

This PR only changes the data source. The CNMF gufunc output_dtypes in cnmf.py still emit float64, and sklearn LassoLars / cvxpy solvers upcast internally, so numeric results shift only as much as a float32 input warrants. Tightening those (gufunc output_dtypes, internal np.zeros dtypes, double accumulators) is a validated follow-up (Phase 2).

Validation

test_pipeline_notebook runs this notebook end-to-end in CI and checks cell count (286 ±10) and the C/S/A sums (rel 5e-2 / 1e-1 / 5e-2) - tolerances built for exactly this kind of numeric drift. That's the gate for the notebook change. The io default change is covered by test_pre_processing (which passes dtype explicitly, so behavior is unchanged there).

Note

Targets master, which is the v2 development branch, so the load_videos default dtype change (a public-API default change suited to a major version) is appropriate here. It's also backward-compatible for 8-bit-sourced data, since those values are exact in float32.

🤖 Generated with Claude Code

Phase 1 of moving minian off float64. Two source-of-truth changes; the rest of
the pipeline infers dtype from its inputs and follows along.

- io.load_videos default dtype np.float64 -> np.float32. The footage is 8-bit,
  so every loaded value is represented exactly in float32; this just stops
  inflating it 8x in memory and on disk. Callers that pass dtype explicitly
  (including the demo notebook and the tests) are unaffected.
- pipeline notebook: Y.astype(float) -> Y.astype(np.float32) at the Y_fm_chk
  save, the single spot where the demo movie was promoted to float64. This is
  the ~69 GB-vs-34 GB cube behind the slow save/rechunk; everything downstream
  (Y_fm_chk, Y_hw_chk, seed init, CNMF prep) is now float32. All other notebook
  casts were already float32.

Scope is deliberately limited to the data source. The CNMF gufunc
output_dtypes (cnmf.py) still emit float64, and sklearn/cvxpy upcast their
solves internally, so numeric results move only as much as a float32 input
warrants. The pipeline notebook test (test_pipeline_notebook) validates that
cell count (286 +/-10) and the C/S/A sums stay within their tolerances.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daharoni
daharoni marked this pull request as draft June 26, 2026 08:22
@daharoni
daharoni marked this pull request as ready for review June 26, 2026 08:41
@codecov

codecov Bot commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@daharoni
daharoni merged commit 45d510a into master Jun 26, 2026
17 checks passed
@daharoni
daharoni deleted the perf/float32-data-path branch June 26, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant