Satip
Satip copied to clipboard
Use Zarr v3 Sharded Store
Detailed Description
Once https://zarr.dev/zeps/draft/ZEP0002.html is finialized and merged into zarr-python, we should use the sharded zarr store to reduce number of files, while making the actual chunks a lot smaller, to minimize extra reads needed.
Context
Should speed up reading more.
can i get your inputs on this issue what should be done now? @jacobbieker