diff --git a/doc/user-guide/io.rst b/doc/user-guide/io.rst index 81fa29bdf5f..bf8b1870572 100644 --- a/doc/user-guide/io.rst +++ b/doc/user-guide/io.rst @@ -776,6 +776,75 @@ dimensions included in ``region``. Other variables (typically coordinates) need to be explicitly dropped and/or written in a separate calls to ``to_zarr`` with ``mode='a'``. +.. _io.zarr.writing_chunks: + +Specifying chunks in a zarr store +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Chunk sizes may be specified in one of three ways when writing to a zarr store: + +1. Manual chunk sizing through the use of the ``encoding`` argument in :py:meth:`Dataset.to_zarr`: +2. Automatic chunking based on chunks in dask arrays +3. Default chunk behavior determined by the zarr library + +The resulting chunks will be determined based on the order of the above list; dask +chunks will be overridden by manually-specified chunks in the encoding argument, +and the presence of either dask chunks or chunks in the ``encoding`` attribute will +supercede the default chunking heuristics in zarr. + +Importantly, this logic applies to every array in the zarr store individually, +including coordinate arrays. Therefore, if a dataset contains one or more dask +arrays, it may still be desirable to specify a chunk size for the coordinate arrays +(for example, with a chunk size of `-1` to include the full coordinate). + +To specify chunks manually using the ``encoding`` argument, provide a nested +dictionary with the structure ``{'variable_or_coord_name': {'chunks': chunks_tuple}}``. + +.. note:: + + The positional ordering of the chunks in the encoding argument must match the + positional ordering of the dimensions in each array. Watch out for arrays with + differently-ordered dimensions within a single Dataset. + +For example, let's say we're working with a dataset with dimensions +``('time', 'x', 'y')``, a variable ``Tair`` which is chunked in ``x`` and ``y``, +and two multi-dimensional coordinates ``xc`` and ``yc``: + +.. ipython:: python + + ds = xr.tutorial.open_dataset("rasm") + + ds["Tair"] = ds["Tair"].chunk({"x": 100, "y": 100}) + + ds + +These multi-dimensional coordinates are only two-dimensional and take up very little +space on disk or in memory, yet when writing to disk the default zarr behavior is to +split them into chunks: + +.. ipython:: python + + ds.to_zarr("path/to/directory.zarr", mode="w") + ! ls -R path/to/directory.zarr + + +This may cause unwanted overhead on some systems, such as when reading from a cloud +storage provider. To disable this chunking, we can specify a chunk size equal to the +length of each dimension by using the shorthand chunk size ``-1``: + +.. ipython:: python + + ds.to_zarr( + "path/to/directory.zarr", + encoding={"xc": {"chunks": (-1, -1)}, "yc": {"chunks": (-1, -1)}}, + mode="w", + ) + ! ls -R path/to/directory.zarr + + +The number of chunks on Tair matches our dask chunks, while there is now only a single +chunk in the directory stores of each coordinate. + .. _io.iris: Iris