Skip to content

perf: avoid per-chunk dtype checks in BytesCodec decoding - #4517

Open
victorwon2001 wants to merge 1 commit into
zarr-developers:mainfrom
victorwon2001:fix/issue-3703
Open

victorwon2001 wants to merge 1 commit into
zarr-developers:mainfrom
victorwon2001:fix/issue-3703

Conversation

@victorwon2001

Copy link
Copy Markdown

Summary

Eliminate per-chunk HasEndianness and Struct type checks from BytesCodec._decode_sync. The stored-byte view now uses the native NumPy dtype with newbyteorder(self.endian) whenever an endian is set, and uses the unchanged dtype otherwise. Byte-swapping, shape restoration and prototype-selected buffers remain intact; the public API and CPU/CUDA abstraction are unchanged.

For reviewers

The change is intended to preserve existing data and byte-order semantics. Direct and store-backed tests now cover integers and floats of multiple widths, mixed-endian structured fields, one-byte data, raw bytes, zero-copy matching-endian decoding and copying only for byte-order conversion. An alternating CPU benchmark found lower per-chunk scalar overhead and similar structured performance; the focused tests and full lint passed, while CUDA could not be checked locally.

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.md
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

Fixes #3703

@github-actions github-actions Bot added the needs release notes Automatically applied to PRs which haven't added release notes label Oct 16, 2026
@codecov

codecov Bot commented Oct 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.72%. Comparing base (bd0dc84) to head (0a0f416).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4517   +/-   ##
=======================================
  Coverage   94.71%   94.72%           
=======================================
  Files          94       94           
  Lines       13663    13659    -4     
=======================================
- Hits        12941    12938    -3     
+ Misses        722      721    -1     
Files with missing lines Coverage Δ
src/zarr/codecs/bytes.py 100.00% <100.00%> (+1.21%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs release notes Automatically applied to PRs which haven't added release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

minimize polymorphism in the codec pipeline?

1 participant