Zarr v3 Support and conda-forge versus pypi dependencies

[scottyhq]

Moving some discussion from #877 (comment) to it's own issue.

Currently the conda-forge feedstock requires Zarr<3 and Virtualizarr>=1.2
https://github.com/conda-forge/earthdata-feedstock/blob/86aaa60fcacaf8a2ae3d8ce608f583b45a7d5fd1/recipe/meta.yaml#L39-L40

    - zarr >=2.12.0, <3.0.0a
    - virtualizarr >=1.2.0

But virtualizarr>=1.3.3 will pin zarr>=3, which I think is going to cause some confusion.
zarr-developers/VirtualiZarr#392
https://github.com/zarr-developers/VirtualiZarr/compare/main..develop#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711

From the pyproject.toml it looks like virtualizarr is actually an optional dependency, so maybe it should just be removed from the conda-forge recipe?https://github.com/nsidc/earthaccess/blob/18e947ec4660ccfe83038ebdd825ed1d08e5990d/pyproject.toml#L57-L64

Alternatively, not sure what work is needed to support Zarr v3 here?

[weiji14]

Thanks @scottyhq for opening this issue to track!

But virtualizarr>=1.3.3 will pin zarr>=3, which I think is going to cause some confusion.

I've checked with Ayush in this Zulip thread, and got his blessing to pin to zarr>=3 (i.e. drop zarr-python=2 compatibility)

From the pyproject.toml it looks like virtualizarr is actually an optional dependency, so maybe it should just be removed from the conda-forge recipe?

Yes, it's usually best to not have opitonal dependencies in a conda recipe (conda/conda#7502), but that should change soon once this CEP is merged (see also https://prefix.dev/blog/ceps_2025#optional-dependencies-conditional-dependencies-and-flags).

Alternatively, not sure what work is needed to support Zarr v3 here?

Working on this at #967. @betolink has kindly given me contributor access during the earthaccess hackathon today, and hopefully we can get this in for the next release.

[mfisher87]

Amazing! I can't believe we're getting optional dependencies soon. 🎉 🎊 🙌 Thanks for sharing this!

[weiji14]

I think this can be closed now since Zarr v3 is used since #1074 (released in earthaccess v0.15.0), and conda-forge/earthdata-feedstock#55 removed the dependency on Zarr (since it is only optional).