Building competence for the eerie.cloud data workflow.
Links:
- Gitlab Repo
- Technical overview paper
- Access and usage:
- Ingest: Lake house approach paper
For the cloudify tarining on levante, start a jupyterhub server on either a compute (recommended) or an interactive node.
- Zarr - a (not only) cloud-optimized data format for ESM output
- Benefits of cloud storages and why we not fully use it (yet)
- Xpublish - the cloud data emulator with server-side processing
- How to start an app
- The various ways to access cloudified data through catalogs, xarray and cdo
- Use-cases and preparations for a data server on the PB scale
- Server-side processing for lossy compression, rechunking and on-the-fly post-processing
- Large aggregations: Zarr becomes the catalog with kerchunked input and the kerchunk API
- SPOA for ingestions
- Requirements for a performant data server
- Kerchunking: we create virtual datasets by extracting the storage chunks of netcdf and grib files, concat them and store the consolidated dataset in a lazy format based on parquet tables.
- Catalogs: The virtual zarr datasets are collected in an intake catalog based on intake-xarray. This catalog is used for eerie.cloud ingestion.
- The openstack VM setting, Nginx and a xpublish plugin.
- The catalog infrastructure based on a mixture of static and dynamic STAC catalogs.
Live show
Navigation through eerie.cloud with static stac catalogs in the web-browser using the stac-browser.
User guide with the easy gems notebook
Applications: Jupyterlite, Gridlook and a WPS based on a simple xarray API
Integrations to be discussed:
- Freva
- Warmworld approach
8.1. 13:00-15:00 hybrid: room #23 or https://eu02web.zoom-x.de/j/9290696892?pwd=WElNS0xIMGp3ZERIRTlYdjR0U3ZaUT09
For DKRZ DM.