Can't import `zarr<3` with latest `numcodecs==0.16.0`

[ryan-williams]

Zarr version

v2.18.5

Numcodecs version

v0.16.0

Python Version

3.11 (but probably all)

Operating System

Ubuntu and Mac (but probably all)

Installation

pip install 'zarr<3' numcodecs

Description

In zarr<3, zarr/util.py imports numcodecs.blosc.cbuffer_sizes (e.g. here in 2.18.5), which was removed in numcodecs 0.16.0 (released 40mins ago at time of writing)

This reference was removed in #2182, prior to 3.0.0, but causes zarr 2.x to fail to import when latest numcodecs is installed; GHA example:

  File "/home/runner/work/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.11/site-packages/zarr/convenience.py", line 8, in <module>
    from zarr._storage.store import data_root, meta_root, assert_zarr_v3_api_available
  File "/home/runner/work/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.11/site-packages/zarr/_storage/store.py", line 10, in <module>
    from zarr.meta import Metadata2, Metadata3
  File "/home/runner/work/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.11/site-packages/zarr/meta.py", line 10, in <module>
    from zarr.util import json_dumps, json_loads
  File "/home/runner/work/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.11/site-packages/zarr/util.py", line 34, in <module>
    from numcodecs.blosc import cbuffer_sizes, cbuffer_metainfo
ImportError: cannot import name 'cbuffer_sizes' from 'numcodecs.blosc' (/home/runner/work/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.11/site-packages/numcodecs/blosc.cpython-311-x86_64-linux-gnu.so)
Error: Process completed with exit code 1.

Not sure it's worth a 2.18.6 to pin numcodecs<0.16 (or remove the offending references), just flagging the issue.

Steps to reproduce

python -m venv .venv
. .venv/bin/activate
pip install 'zarr<3' numcodecs
python -c 'import zarr'  # ❌ ImportError: cannot import name 'cbuffer_sizes' from 'numcodecs.blosc' (/Users/ryan/c/single-cell-data/tiledb-soma/tmp/.venv/lib/python3.11/site-packages/numcodecs/blosc.cpython-311-darwin.so)

or:

docker run --rm --entrypoint bash python:3.11 -c 'pip install "zarr<3" numcodecs && python -c "import zarr"'
# ❌ ImportError: cannot import name 'cbuffer_sizes' from 'numcodecs.blosc' (/usr/local/lib/python3.11/site-packages/numcodecs/blosc.cpython-311-x86_64-linux-gnu.so)

Additional output

No response

[dstansby]

Thanks for the quick report, and sorry about this. I think the best thing to do is pin numcodecs here, I'll do that now.

[cspades]

Also hit this error, which affects NeMo:

/__w/_tool/Python/3.12.9/x64/lib/python3.12/site-packages/zarr/meta.py:10: in <module>
    from zarr.util import json_dumps, json_loads
/__w/_tool/Python/3.12.9/x64/lib/python3.12/site-packages/zarr/util.py:34: in <module>
    from numcodecs.blosc import cbuffer_sizes, cbuffer_metainfo
E   ImportError: cannot import name 'cbuffer_sizes' from 'numcodecs.blosc' (/__w/_tool/Python/3.12.9/x64/lib/python3.12/site-packages/numcodecs/blosc.cpython-312-x86_64-linux-gnu.so)

Following for the fix, thank you!

[dstansby]

This should now be fixed with a new 2.18.6 release - would be great if someone could confirm before we close this.