Enable deterministic and formatted metadata output

[lkluft]

Currently, the order of entries in Zarr metadata is non-deterministic. This was previously discussed in issue #3254, where it was concluded that the Zarr specification does not prescribe a specific order—so downstream applications should not rely on one.

However, it would still be beneficial if writing the same dataset consistently produced identical metadata. This would, for example, facilitate computing dataset checksums.

Additionally, Zarr v3 now writes unformatted JSON to the .zmetadata file of v2 stores. In Zarr v2, human-readable (i.e., pretty-printed) metadata was preferred (see #704), a practice that v3 also adopted (see #1952). I assume that this is simply a regression. Ideally, metadata formatting would be consistent across versions and backends.

It would be great to consider enabling deterministic and formatted metadata output. This would improve reproducibility, readability, and tooling interoperability without conflicting with the specification.

/cc @d70-t

[d-v-b]

this makes sense, I see two actionable items here:

• we ensure that the keys of the metadata dict are written in a consistent order (ideally the order in which the keys are defined in the zarr spec)
• We ensure that writing .zmetadata is sensitive to the global JSON formatting configuration.

evidently we were not testing either of these behaviors

[d-v-b]

@lkluft can you link to any test failures caused by these regressions, or point me to the code that's sensitive to key order?

[lkluft]

The following code snippet is sufficient to reproduce both issues:

import xarray as xr


ds = xr.Dataset({"a": range(10), "b": range(10)})

# Creates a "non-formatted" store_v2.zarr/.zmetadata
ds.to_zarr("store_v2.zarr", zarr_format=2, mode="w")

# Creates a pretty-formatted store_v3.zarr/zarr.json
ds.to_zarr("store_v3.zarr", zarr_format=3, mode="w")

This directly shows that the v2 metadata is not formatted, whereas the v3 metadata is. (In the v2 library, the v2 metadata was formatted.)

The order of the keys is more subtle. While all Zarr-specific (top-level) keys appear to be deterministic, the order of the variables in the consolidated metadata is not. Here, "a" and "b" may appear in any order if the snippet is run repeatedly. It seems that the consolidation of the metadata occurs after the variables have been written. Consequently, a non-deterministic listing of the filesystem could cause this behaviour. However, sorting the consolidated metadata before attaching it should solve this.

[d-v-b]

ok maybe I misunderstood the scope of the term "metadata". I thought this issue was about array metadata documents, but it looks like this is pretty specific to consolidated metadata, is that right?

[lkluft]

ok maybe I misunderstood the scope of the term "metadata". I thought this issue was about array metadata documents, but it looks like this is pretty specific to consolidated metadata, is that right?

Yes, I wasn't very specific on that. So far, this seems to only affect the consolidated metadata. However, this still implies the above mentioned problems of reproducibility (in terms of checksums).

[d-v-b]

if you're interested in contributing a patch, I think we could add a step that sorts the contents of consolidated metadata. The sorting algorithm should probably sort by ascending path length, where a path is a sequence of strings joined by the "/", falling back to lexicographic sorting for keys with the same path length.

[lkluft]

Sure! I opened #3288 with a straight-forward implementation. I'm not sure if this approach is too simplistic, but I'm happy to discuss other options.

[d-v-b]

thank you! I will look at it tomorrow, unless another dev gets there first :)