echopype icon indicating copy to clipboard operation
echopype copied to clipboard

Add `'auto'` option to kwarg `offload_to_zarr` in `open_raw`

Open b-reyes opened this issue 3 years ago • 0 comments

In PR #774 we added the option to directly write variables with a large memory footprint to a temporary zarr store. Ideally, we want to include the option offload_to_zarr='auto', which will trigger a routine that automatically determines if we should directly write these variables to a temporary zarr. In a first attempt, the choice of whether to offload to zarr was based on a threshold of the percentage of memory consumed, if the variables were to be expanded. However, as highlighted by @lsetiawan's comment, it may be difficult to make this robust for all computer architectures and take into account the memory increase downstream.

One alternative to this approach would be to write those variables that when expanded exceed the user input value max_zarr_mb. The input max_zarr_mb is the maximum MB that each zarr chunk should hold, when offloading variables with a large memory footprint to a temporary zarr store.

b-reyes avatar Aug 11 '22 22:08 b-reyes