#2656 introduced time windowing to Parcels. This greatly improved performance when working with larger-than-memory datasets as it eliminate's Dask overhead on the vectorized indexing level. However, there are still improvements that can be made:
- to reduce the memory overhead: at the moment we're loading whole timeslices which has significant memory use when the particles may only be in a small portion of the domain
- to improve integration with Xarray: currently we're working completely in Parcels - at a level of abstraction above Xarray. This is a somewhat messy abstraction level to work at, resulting in maintainer burden, and I think it can be put within the underlying arrays within Xarray instead.
Something like
I've made significant progress on an implementation, which (from prelim testing on my laptop) has significantly better performance than windowed array up to 1M particles. Just wanting to do some further work to make sure the abstractions work well with Xarray (and that it's properly tested), before making a PR.
Checklist:
#2656 introduced time windowing to Parcels. This greatly improved performance when working with larger-than-memory datasets as it eliminate's Dask overhead on the vectorized indexing level. However, there are still improvements that can be made:
Something like
I've made significant progress on an implementation, which (from prelim testing on my laptop) has significantly better performance than windowed array up to 1M particles. Just wanting to do some further work to make sure the abstractions work well with Xarray (and that it's properly tested), before making a PR.
Checklist:
FieldSet.use_backend("windowed_arrays"),FieldSet.use_backend("chunked_arrays")or similar for the API? I think that would be a clearer API than having a bunch of methods.