Skip to content

Allow disabling compression in versions 1.2.x (Zarr v3), and document it #584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
FedeMPouzols wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Allow disabling compression in versions 1.2.x (Zarr v3), and document it #584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/xradio/measurement_set/_utils/_msv2/conversion.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1012,7 +1012,7 @@ def convert_and_write_partition(
ephemeris_interpolate: bool = False,
phase_cal_interpolate: bool = False,
sys_cal_interpolate: bool = False,
compressor: zarr.abc.codec.BytesBytesCodec = zarr.codecs.ZstdCodec(level=2),
compressor: zarr.abc.codec.BytesBytesCodec | None = zarr.codecs.ZstdCodec(level=2),
add_reshaping_indices: bool = False,
storage_backend="zarr",
parallel_mode: str = "none",
Expand Down Expand Up @@ -1503,7 +1503,7 @@ def antenna_ids_to_names(
xds = xds.assign_coords({"antenna_name": xds["baseline_id"].data})
xds = xds.drop_vars("baseline_id")

# drop more vars that seem unwanted in main_sd_xds, but there shouuld be a better way
# drop more vars that seem unwanted in main_sd_xds, but there should be a better way
# of not creating them in the first place
unwanted_coords_sd = ["baseline_antenna1_id", "baseline_antenna2_id"]
for unwanted_coord in unwanted_coords_sd:
Expand Down
3 changes: 2 additions & 1 deletion src/xradio/measurement_set/_utils/_zarr/encoding.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@ def add_encoding(xds, compressor, chunks=None):

for da_name in list(xds.data_vars):
da_chunks = [chunks[dim_name] for dim_name in xds[da_name].sizes]
xds[da_name].encoding = {"compressors": (compressor,), "chunks": da_chunks}
compressors = (compressor,) if compressor else ()
xds[da_name].encoding = {"compressors": compressors, "chunks": da_chunks}
4 changes: 2 additions & 2 deletions src/xradio/measurement_set/convert_msv2_to_processing_set.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ def convert_msv2_to_processing_set(
phase_cal_interpolate: bool = False,
sys_cal_interpolate: bool = False,
use_table_iter: bool = False,
compressor: zarr.abc.codec.BytesBytesCodec = zarr.codecs.ZstdCodec(level=2),
compressor: zarr.abc.codec.BytesBytesCodec | None = zarr.codecs.ZstdCodec(level=2),
add_reshaping_indices: bool = False,
storage_backend: Literal["zarr", "netcdf"] = "zarr",
parallel_mode: Literal["none", "partition", "time"] = "none",
Expand Down Expand Up @@ -114,7 +114,7 @@ def convert_msv2_to_processing_set(
use_table_iter : bool, optional
Whether to use the table iterator to read the main table of the MS v2. This should be set to True when reading datasets with large number of rows and few partitions, by default False.
compressor : numcodecs.abc.Codec, optional
The Blosc compressor to use when saving the converted data to disk using Zarr, by default numcodecs.Zstd(level=2).
The compressor to use when saving the converted data to disk using Zarr, by default numcodecs.Zstd(level=2). None disables compression.
add_reshaping_indices : bool, optional
Whether to add the tidxs, bidxs and row_id variables to each partition of the main dataset. These can be used to reshape the data back to the original ordering in the MS v2. This is mainly intended for testing and debugging, by default False.
storage_backend : Literal["zarr", "netcdf"], optional
Expand Down
Loading