Skip to content

chunks #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alxmrs merged 2 commits into from
Aug 4, 2024
Merged

chunks #56

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e33df03
chunnks
RichardScottOZ Aug 3, 2024
7a242d7
space
RichardScottOZ Aug 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ That's it!
## Why does this work?

Underneath Xarray, Dask, and Pandas, there are NumPy arrays. These are paged in
chuncks and represented contiguously in memory. It is only a matter of metadata
chunks and represented contiguously in memory. It is only a matter of metadata
that breaks them up into ndarrays. `to_dataframe()`
just changes this metadata (via a `ravel()`/`reshape()`), back into a column
amenable to a DataFrame.
Expand Down Expand Up @@ -100,7 +100,7 @@ Xarray Datasets. This approach is being pursued
Deeper still: I was thinking we could make
a [virtual](https://fsspec.github.io/kerchunk/)
filesystem for parquet that would internally map to Zarr. Raster-backed virtual
parquet would open up integrations to numeroustools like dask, pyarrow, duckdb,
parquet would open up integrations to numerous tools like dask, pyarrow, duckdb,
and BigQuery. More thoughts on this
in [#4](https://github.com/alxmrs/xarray-sql/issues/4).

Expand Down