Provide informative errors when opening Icechunk data without the Icechunk storage transformer

[maxrjones]

It would be better to provide a useful error message when users try to access Icechunk stores via Zarr directly.

# /// script
# dependencies = [
#     "zarr",
#     "obstore",
# ]
# ///

from zarr.storage import ObjectStore
from zarr import open_array
from obstore.store import HTTPStore

url = "https://data.dynamical.org/ecmwf/ifs-ens/forecast-15-day-0-25-degree/latest.zarr"
zarr_store = ObjectStore(HTTPStore.from_url(url))
temperature_2m = open_array(
    zarr_store,
    path="temperature_2m"
)
print(temperature_2m)

print(temperature[0,0,0])

$ uv run --script icechunk.py
<xarray.Dataset> Size: 250TB
Dimensions:                                     (init_time: 732, lead_time: 85,
                                                 ensemble_member: 51,
                                                 latitude: 721, longitude: 1440)
Coordinates:
  * init_time                                   (init_time) datetime64[ns] 6kB ...
    expected_forecast_length                    (init_time) timedelta64[ns] 6kB ...
    ingested_forecast_length                    (init_time) timedelta64[ns] 6kB ...
  * lead_time                                   (lead_time) timedelta64[ns] 680B ...
    valid_time                                  (init_time, lead_time) datetime64[ns] 498kB ...
  * ensemble_member                             (ensemble_member) int16 102B ...
  * latitude                                    (latitude) float64 6kB 90.0 ....
  * longitude                                   (longitude) float64 12kB -180...
    spatial_ref                                 int64 8B ...
Data variables: (12/19)
    ...
Traceback (most recent call last):
  File "icechunk.py", line 23, in <module>
    t2_forecast.plot()
  ...
  File ".../zarr/codecs/crc32c_.py", line 49, in _decode_sync
    raise ValueError(
ValueError: Stored and computed checksum do not match. Stored: b'i\x05\xb1\xec'. Computed: b'h\xd495'.

[d-v-b]

Is this specific to icechunk, or does it generalize to using store X to open data saved with store Y? In this case, store X is obstore's http backend, and store Y is icechunk. Kind of interesting that they use overlapping chunk keys though. And how did you manage to open zarr.json? Is it possible that you created a new array?

in any case we should handle this ValueError by telling the user the name of the chunk that couldn't be opened, and how we tried to open it.

[maxrjones]

those are great questions. I'd be super grateful if anyone is interested in devoting some time to finding the answers. I was also surprised by the error and the workflow apparently being fixed by using an Icechunk store.

[aldenks]

I think this is not a zarr-python issue and rather an issue specific to that proxy. I happen to know that https://data.dynamical.org has never served up Icechunk data, only standard Zarr v3. I never got to the bottom of it and we have deprecated the proxy (for this and other reasons) but I did occasionally see the same error you saw @maxrjones.