Refreshed the documentation

README.md

@@ -11,36 +11,72 @@ pip install xarray-sql
 
 ## What is this?
 
-This is an experiment to provide a SQL interface for raster data.
+This is an experiment to provide a SQL interface for array datasets.
 
 ```python
 import xarray as xr
-import xarray_sql as qr
+import xarray_sql as xql
 
 ds = xr.tutorial.open_dataset('air_temperature')
 
 # The same as a dask-sql Context; i.e. an Apache DataFusion Context.
-c = qr.XarrayContext()
-c.from_dataset('air', ds, chunks=dict(time=24))
-
-df = c.sql('''
+ctx = xql.XarrayContext()
+ctx.from_dataset('air', ds, chunks=dict(time=24))  # the Dataset needs to be chunked!
+# DataFrame()
+# +------+---------------------+-------+--------------------+
+# | lat  | time                | lon   | air                |
+# +------+---------------------+-------+--------------------+
+# | 75.0 | 2013-01-01T00:00:00 | 200.0 | 241.20000000000002 |
+# | 75.0 | 2013-01-01T00:00:00 | 202.5 | 242.5              |
+# | 75.0 | 2013-01-01T00:00:00 | 205.0 | 243.5              |
+# | 75.0 | 2013-01-01T00:00:00 | 207.5 | 244.0              |
+# | 75.0 | 2013-01-01T00:00:00 | 210.0 | 244.1              |
+# | 75.0 | 2013-01-01T00:00:00 | 212.5 | 243.89000000000001 |
+# | 75.0 | 2013-01-01T00:00:00 | 215.0 | 243.6              |
+# | 75.0 | 2013-01-01T00:00:00 | 217.5 | 243.1              |
+# | 75.0 | 2013-01-01T00:00:00 | 220.0 | 242.5              |
+# | 75.0 | 2013-01-01T00:00:00 | 222.5 | 241.89000000000001 |
+# +------+---------------------+-------+--------------------+
+# Data truncated.
+
+result = ctx.sql('''
   SELECT
     "lat", "lon", AVG("air") as air_total
   FROM 
     "air" 
   GROUP BY
    "lat", "lon"
 ''')
+# DataFrame()
+# +------+-------+--------------------+
+# | lat  | lon   | air_total          |
+# +------+-------+--------------------+
+# | 75.0 | 205.0 | 259.88662671232834 |
+# | 75.0 | 207.5 | 259.48268150684896 |
+# | 75.0 | 230.0 | 258.9192123287667  |
+# | 75.0 | 275.0 | 257.07574315068456 |
+# | 75.0 | 322.5 | 250.11792123287654 |
+# | 75.0 | 325.0 | 250.81590068493125 |
+# | 72.5 | 205.0 | 262.74933904109537 |
+# | 72.5 | 207.5 | 262.5384315068488  |
+# | 72.5 | 230.0 | 260.8287945205475  |
+# | 72.5 | 275.0 | 257.30633219178037 |
+# +------+-------+--------------------+
+# Data truncated.
+# 
 
 # A table of the average temperature for each location across time.
-df.to_pandas()
-
-# Alternatively, you can just create the DataFrame from the RecordBatchReader:
-df = qr.read_xarray(ds).read_pandas()
+df = result.to_pandas()
 df.head()
+#     lat    lon   air_total
+# 0  75.0  210.0  259.016562
+# 1  75.0  222.5  258.362212
+# 2  75.0  237.5  258.318240
+# 3  75.0  267.5  256.928497
+# 4  75.0  285.0  261.614103
 ```
 
-Succinctly, we "pivot" Xarray Datasets to treat them like tables so we can run
+Succinctly, we "pivot" Xarray Datasets (with consistent dimensions) to treat them like tables so we can run
 SQL queries against them.
 
 ## Why build this?
@@ -64,7 +100,7 @@ front end.
 
 ## How does it work?
 
-All chunks in an Xarray Dataset are transformed into a Dask DataFrame via
+All chunks in a Xarray Dataset are transformed into a Dask DataFrame via
 `from_map()` and `to_dataframe()`. For SQL support, we just use `dask-sql`.
 That's it!
 
@@ -77,14 +113,15 @@ Underneath Xarray, Dask, and Pandas, there are NumPy arrays. These are paged in
 chunks and represented contiguously in memory. It is only a matter of metadata
 that breaks them up into ndarrays. `to_dataframe()`
 just changes this metadata (via a `ravel()`/`reshape()`), back into a column
-amenable to a DataFrame.
-
-There is added overhead from duplicating dimensions as columns, which we see as
-worth the convenience of DataFrames.
+amenable to a DataFrame. This project basically just makes a metadata change to 
+make this chunked information scannable by a DB engine (DataFusion).
 
 ## What are the current limitations?
 
-_2025 update_: TBD, `datafusion` provides a whole new world!
+_2025 update_: TBD, `datafusion` provides a whole new world! Currently, this project is looking
+for early users – "tire kickers", if you will. We'd love your input to shape the direction of this 
+project! Please, give this a try and [file issues](https://github.com/alxmrs/xarray-sql/issues) as
+you see fit. Check out our [contributing guide](CONTRIBUTING.md), too 😉.
 
 ## What would a deeper integration look like?
 
@@ -99,6 +136,13 @@ parquet would open up integrations to numerous tools like dask, pyarrow, duckdb,
 and BigQuery. More thoughts on this
 in [#4](https://github.com/alxmrs/xarray-sql/issues/4).
 
+_2025 update_: Something like this is being built across a few projects! The ones I know about are:
+- [CartoDB's Raquet](https://github.com/CartoDB/raquet)
+- The DataFusion community's [arrow-zarr](https://github.com/datafusion-contrib/arrow-zarr)
+
+As of writing, this project is [amid integrating](https://github.com/alxmrs/xarray-sql/pull/69) a
+rust-based DataFusion backend provided by arrow-zarr.
+
 ## Sponsors & Contributors
 
 I want to give a special thanks to the following folks and institutions:
@@ -108,6 +152,9 @@ I want to give a special thanks to the following folks and institutions:
 - Jake Wall and AI2/Ecoscope for compute resources and key use cases.
 - Charles Stern, Stephan Hoyer, Alexander Kmoch, Wei Ji, and Qiusheng Wu
   for the early review and discussion of this project.
+- Tom Nichols, Kyle Barron, Tom White, and Maxime Dion for the [Array Working
+  Group](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734)
+  and DataFusion-specific collaboration.
 
 ## License