DOC: `LRUStoreCache` removed in Zarr 3.0, not documented

[ljwoods2]

Describe the issue linked to the documentation

I may be missing something, but noticed our CI was failing with Zarr v3 but can't find any deprecation warning or migration information for the zarr.storage.LRUStoreCache.

Is this no longer supported, or is there another way to do this?

Suggested fix for documentation

No response

[d-v-b]

sorry about that! looks like we forgot to add a deprecation to LRUStoreCache.

We have not implemented this for v3. In the longer term, I think we see caching as an important part of basic storage performance, and when we have have caching baked in to all stores, then a specialized LRUStoreCache store wouldn't make much sense.

But our future plans are not much use to you today. To get LRUStoreCache functionality going right now, I would look at the general purpose store wrapper class. If you subclassed WrapperStore and overrode __init__ to add cache setup logic, and overrode get(key) to add "check if key is in the cache, otherwise fetch it" logic then I think everything would work.

Let me know if that turns out to be bad advice, or you want more pointers.

[mtvector]

Hey sorry to just jump in on this thread, but I am also interested in a simple caching system for an Xarray backend, where we just keep the most recently used chunk for each variable (would help a lot for sequentially iterating through minibatches) I've tried many versions along the lines of the following:

class LastChunkCacheStore(WrapperStore):
    def __init__(self, store):
        super().__init__(store)
        self._cache = {} #should actually track variable+chunk

    async def get(self, key: str, prototype, byte_range=None):
        if byte_range is None and key in self._cache:
            return self._cache[key]
        value = await self._store.get(key, prototype, byte_range)
        if byte_range is None:
            self._cache[key] = value
            #dump previous chunk once I get it working
        return value

But inevitably I'm running into errors like RuntimeError: cannot reuse already awaited coroutine, or if I try to make it synchronous I get things like SyncError: Calling sync() from within a running loop. Maybe it's not as simple as I imagined...

If you have more advice about how to go about this or a simple example to work off of, that would be amazing (I'm a real beginner when it comes to zarr and I/O)
Thanks!

[d-v-b]

@mtvector i feel like what you are trying should work. if I have some time this week I can play around with this problem and try to get something working (or figure out why it's not working)

[rabernat]

It's not super well documented, but the Icechunk storage engine implements a pretty powerful caching system.

IMO, one problem with the earlier iterations of Zarr caching is that there is a lot of ambiguity about what can be cached and for how long. This is not a problem for static datasets, but for "live" datasets that get updates, it can lead to all kinds of challenges about cache eviction, etc.

Icechunk overcomes those issues by introducing the concept of snapshots and by using immutable identifiers for both chunks and metadata. The immutable identifiers can safely be cached forever. And the cache is managed from the Rust side, avoiding the sort of sync challenges you experienced above.

@mtvector - would love to see if Icechunk would work for your use case. Let me know if you'd like help getting started, and feel free to ask questions at https://github.com/earth-mover/icechunk/discussions.

[mtvector]

Thanks for this, I'll try out Icechunk and post back with progress or problems!