Xarray-specific filter predicate pushdown.

[alxmrs]

Partition pruning: In the Xarray data model, we can always get statistics for the dataset dims, which function like the "primary keys" of the materialized table. These statistics lend themselves to omit processing data that we know we don't need to. For example,

SELECT * FROM air WHERE time > '2020-02-01'

Will skip loading partitions whose time ranges are entirely before 2020-02-01.

[alxmrs]

@claude can you review this PR?

[maximedion2]

Just for context, so I see that this is to push down filter predicates to avoid reading some data, but what is a partition in this context? Like this is to prune partitions, but how do you define it? Is it related to a zarr group (sort of like a hive style partition) or just chunking up the data? I think in the earlier PR I looked at it was just data chunks, but I see you've been making a few changes and I'm trying to keep track of the overall features and logic.

Also, obviously don't wait on me if you want to merge, I'll take a look either way, this is very interesting, and if I have any comments after you merge I'll still make them here.

[alxmrs]

I'm happy to get your review when you can provide it, @maximedion2. I probably will go ahead and merge now.

what is a partition in this context?

A partition is one chunk of xarray data across multiple variables. Since the ideal chunks are the underlying physical chunks from the Zarr store, I've called these "blocks", to borrow from the Zarr library parlance.

Happy to talk more about it in the future. Thanks for taking a look.