expose methods on python store classes

[d-v-b]

consider exposing some methods on the store classes. Without being able to inspect the state of the storage backend, it's difficult to know the state of an array -- has the metadata document been written, which chunks exist, etc.

[kylebarron]

We don't currently have any store classes. We have instances or protocols that we accept, but no store or async store object.

I think store methods is the domain of obstore/obspec. To expose store methods here would be to literally re-implement obstore.

[d-v-b]

so what's the recommended workflow for inspecting the stored state of an array?

[kylebarron]

stored state

What kind of state? Like whether a chunk exists?

As far as I can tell, there's nothing specific in zarrs that defines stuff like that. It would be up to you to use the store input directly to check that.

Or perhaps this is covered with Node or Hierarchy?

[d-v-b]

What kind of state? Like whether a chunk exists?

yeah, whether a chunk exists, how big it is, and also the state of the written array metadata document. Array.store is defined, but it doesn't have any methods. Unless i'm missing something, this means an array consumer needs to consume (array, store) pairs if they want the option of inspecting the stored state of the array. adding array-typed IO operations could mitigate this -- by design an array could only see its metadata document, and its chunks

[d-v-b]

This is coming up via this example:

from zarr_metadata import ArrayMetadataV3

import zarrista

store = zarrista.store.MemoryStore()

meta_1: ArrayMetadataV3 = {
    "zarr_format": 3,
    "attributes": {},
    "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}},
    "data_type": "int16",
    "chunk_key_encoding": {"name": "default", "configuration": {"separator": "/"}},
    "fill_value": 0,
    "node_type": "array",
    "shape": (4, 4),
    "codecs": ({"name": "bytes"},),
}
meta_2 = {**meta_1, "data_type": "float64"}
arr_1 = zarrista.Array.from_metadata(meta_1, store)
arr_2 = zarrista.Array.from_metadata(meta_2, store)

I'd like to see what's inside the store, since I apparently created 2 arrays at the same spot.

 

[kylebarron]

meta_1: ArrayMetadataV3 = {
    "zarr_format": 3,
    "attributes": {},
    "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}},
    "data_type": "int16",
    "chunk_key_encoding": {"name": "default", "configuration": {"separator": "/"}},
    "fill_value": 0,
    "node_type": "array",
    "shape": (4, 4),
    "codecs": ({"name": "bytes"},),
}
meta_2 = {**meta_1, "data_type": "float64"}
arr_1 = zarrista.Array.from_metadata(meta_1, store)
arr_2 = zarrista.Array.from_metadata(meta_2, store)

I'd like to see what's inside the store, since I apparently created 2 arrays at the same spot.

First, zarrista.Array.from_metadata doesn't create/write metadata. That happens either through ArrayBuilder.create or with Array::store_metadata which we need to bind to Python apparently. #87

We should improve docs that Array.from_metadata does not mutate/write metadata, and improve docs that ArrayBuilder.create does write metadata #88

[d-v-b]

yeah the key question is when metadata is written relative to the life cycle of the Array object

[kylebarron]

Array.store is defined, but it doesn't have any methods.

It is? In Python? I don't think we have a .store attribute?

yeah, whether a chunk exists, how big it is, and also the state of the written array metadata document.
adding array-typed IO operations could mitigate this -- by design an array could only see its metadata document, and its chunks

I think this could be in scope

[kylebarron]

yeah the key question is when metadata is written relative to the life cycle of the Array object

I created new issues for this part, and we can keep this issue to track whether to have store-like methods exposed

[d-v-b]

It is? In Python? I don't think we have a .store attribute?

  haha it's not -- I think I was just inspecting the store object without checking if it was actually bound to the array

[d-v-b]

so given that there's no store attribute, how can I distinguish 2 arrays with the same metadata, and the same path, on different stores?

[kylebarron]

Can you use ==?

[d-v-b]

inequality will indicate when 2 array objects are different, but it's not specific:

(Pdb) zarrista.Array.from_metadata(meta_1, store) == zarrista.Array.from_metadata(meta_1, store)
False

but more broadly, I feel like the array should have a user-facing API for indicating where chunks / metadata get written to.