this issue tracks discussion about / progress towards wrapping compiled Zarr libraries with Zarr-Python
why
go fast
how
we define an API based on a few types:
- JSON
- references to contiguous memory
- simple python types as needed
For example, something like:
read_chunk_into(
array_metadata: JSON,
store_declaration: JSON,
chunk_id: str,
selection: JSON | array_ref,
output: array_ref) -> ReadResult[None]:
# read a chunk, parametrized by a selection, into a pre-allocated array
I think a suite of functions like this would be a solid interface for calling out to compiled zarr-aware tooling. Biggest pain point for this today would be the store: our store classes are not in general JSON serializable. We might need to add a to_json method for stores that can serialize to JSON, and restrict this API to just those stores.
conceptually, this is a deeply obvious move for us to make. the entire point of the zarr spec is to declare a language-agnostic procedure for encoding and decoding chunks. we should use that fact.
this issue tracks discussion about / progress towards wrapping compiled Zarr libraries with Zarr-Python
why
go fast
how
we define an API based on a few types:
For example, something like:
I think a suite of functions like this would be a solid interface for calling out to compiled zarr-aware tooling. Biggest pain point for this today would be the store: our store classes are not in general JSON serializable. We might need to add a
to_jsonmethod for stores that can serialize to JSON, and restrict this API to just those stores.conceptually, this is a deeply obvious move for us to make. the entire point of the zarr spec is to declare a language-agnostic procedure for encoding and decoding chunks. we should use that fact.