From 9968d53a97927ece50e00048473da38227de8e30 Mon Sep 17 00:00:00 2001 From: Alex Merose Date: Sat, 27 Sep 2025 19:32:42 +0200 Subject: [PATCH 1/2] Refreshed README. --- README.md | 80 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 63 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 3747b98a..f5658d54 100644 --- a/README.md +++ b/README.md @@ -11,19 +11,35 @@ pip install xarray-sql ## What is this? -This is an experiment to provide a SQL interface for raster data. +This is an experiment to provide a SQL interface for array datasets. ```python import xarray as xr -import xarray_sql as qr +import xarray_sql as xql ds = xr.tutorial.open_dataset('air_temperature') # The same as a dask-sql Context; i.e. an Apache DataFusion Context. -c = qr.XarrayContext() -c.from_dataset('air', ds, chunks=dict(time=24)) - -df = c.sql(''' +ctx = xql.XarrayContext() +ctx.from_dataset('air', ds, chunks=dict(time=24)) # the Dataset needs to be chunked! +# DataFrame() +# +------+---------------------+-------+--------------------+ +# | lat | time | lon | air | +# +------+---------------------+-------+--------------------+ +# | 75.0 | 2013-01-01T00:00:00 | 200.0 | 241.20000000000002 | +# | 75.0 | 2013-01-01T00:00:00 | 202.5 | 242.5 | +# | 75.0 | 2013-01-01T00:00:00 | 205.0 | 243.5 | +# | 75.0 | 2013-01-01T00:00:00 | 207.5 | 244.0 | +# | 75.0 | 2013-01-01T00:00:00 | 210.0 | 244.1 | +# | 75.0 | 2013-01-01T00:00:00 | 212.5 | 243.89000000000001 | +# | 75.0 | 2013-01-01T00:00:00 | 215.0 | 243.6 | +# | 75.0 | 2013-01-01T00:00:00 | 217.5 | 243.1 | +# | 75.0 | 2013-01-01T00:00:00 | 220.0 | 242.5 | +# | 75.0 | 2013-01-01T00:00:00 | 222.5 | 241.89000000000001 | +# +------+---------------------+-------+--------------------+ +# Data truncated. + +result = ctx.sql(''' SELECT "lat", "lon", AVG("air") as air_total FROM @@ -31,16 +47,36 @@ df = c.sql(''' GROUP BY "lat", "lon" ''') +# DataFrame() +# +------+-------+--------------------+ +# | lat | lon | air_total | +# +------+-------+--------------------+ +# | 75.0 | 205.0 | 259.88662671232834 | +# | 75.0 | 207.5 | 259.48268150684896 | +# | 75.0 | 230.0 | 258.9192123287667 | +# | 75.0 | 275.0 | 257.07574315068456 | +# | 75.0 | 322.5 | 250.11792123287654 | +# | 75.0 | 325.0 | 250.81590068493125 | +# | 72.5 | 205.0 | 262.74933904109537 | +# | 72.5 | 207.5 | 262.5384315068488 | +# | 72.5 | 230.0 | 260.8287945205475 | +# | 72.5 | 275.0 | 257.30633219178037 | +# +------+-------+--------------------+ +# Data truncated. +# # A table of the average temperature for each location across time. -df.to_pandas() - -# Alternatively, you can just create the DataFrame from the RecordBatchReader: -df = qr.read_xarray(ds).read_pandas() +df = result.to_pandas() df.head() +# lat lon air_total +# 0 75.0 210.0 259.016562 +# 1 75.0 222.5 258.362212 +# 2 75.0 237.5 258.318240 +# 3 75.0 267.5 256.928497 +# 4 75.0 285.0 261.614103 ``` -Succinctly, we "pivot" Xarray Datasets to treat them like tables so we can run +Succinctly, we "pivot" Xarray Datasets (with consistent dimensions) to treat them like tables so we can run SQL queries against them. ## Why build this? @@ -64,7 +100,7 @@ front end. ## How does it work? -All chunks in an Xarray Dataset are transformed into a Dask DataFrame via +All chunks in a Xarray Dataset are transformed into a Dask DataFrame via `from_map()` and `to_dataframe()`. For SQL support, we just use `dask-sql`. That's it! @@ -77,14 +113,15 @@ Underneath Xarray, Dask, and Pandas, there are NumPy arrays. These are paged in chunks and represented contiguously in memory. It is only a matter of metadata that breaks them up into ndarrays. `to_dataframe()` just changes this metadata (via a `ravel()`/`reshape()`), back into a column -amenable to a DataFrame. - -There is added overhead from duplicating dimensions as columns, which we see as -worth the convenience of DataFrames. +amenable to a DataFrame. This project basically just makes a metadata change to +make this chunked information scannable by a DB engine (DataFusion). ## What are the current limitations? -_2025 update_: TBD, `datafusion` provides a whole new world! +_2025 update_: TBD, `datafusion` provides a whole new world! Currently, this project is looking +for early users – "tire kickers", if you will. We'd love your input to shape the direction of this +project! Please, give this a try and [file issues](https://github.com/alxmrs/xarray-sql/issues) as +you see fit. Check out our [contributing guide](CONTRIBUTING.md), too 😉. ## What would a deeper integration look like? @@ -99,6 +136,13 @@ parquet would open up integrations to numerous tools like dask, pyarrow, duckdb, and BigQuery. More thoughts on this in [#4](https://github.com/alxmrs/xarray-sql/issues/4). +_2025 update_: Something like this is being built across a few projects! The ones I know about are: +- [CartoDB's Raquet](https://github.com/CartoDB/raquet) +- The DataFusion community's [arrow-zarr](https://github.com/datafusion-contrib/arrow-zarr) + +As of writing, this project is [amid integrating](https://github.com/alxmrs/xarray-sql/pull/69) a +rust-based DataFusion backend provided by arrow-zarr. + ## Sponsors & Contributors I want to give a special thanks to the following folks and institutions: @@ -108,6 +152,8 @@ I want to give a special thanks to the following folks and institutions: - Jake Wall and AI2/Ecoscope for compute resources and key use cases. - Charles Stern, Stephan Hoyer, Alexander Kmoch, Wei Ji, and Qiusheng Wu for the early review and discussion of this project. +- Kyle Barron, Tom Nichols, Tom White, and Maxime Dion for the Array Working + Group and DataFusion-specific discussions. ## License From d7464bd4fb8e55b1485628328be30a03ccfe045d Mon Sep 17 00:00:00 2001 From: Alex Merose Date: Sat, 27 Sep 2025 21:15:17 +0200 Subject: [PATCH 2/2] Some nits to the contributors section --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f5658d54..c57bbe48 100644 --- a/README.md +++ b/README.md @@ -152,8 +152,9 @@ I want to give a special thanks to the following folks and institutions: - Jake Wall and AI2/Ecoscope for compute resources and key use cases. - Charles Stern, Stephan Hoyer, Alexander Kmoch, Wei Ji, and Qiusheng Wu for the early review and discussion of this project. -- Kyle Barron, Tom Nichols, Tom White, and Maxime Dion for the Array Working - Group and DataFusion-specific discussions. +- Tom Nichols, Kyle Barron, Tom White, and Maxime Dion for the [Array Working + Group](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734) + and DataFusion-specific collaboration. ## License