xbatcher
xbatcher copied to clipboard
xarray-contrib
Batch generation from xarray datasets
### What is your issue? Recent developments by @nvidia and @dcherian are opening the door for direct-to-gpu data loading in Xarray. This could mean that when combined with Xbatcher and...
Xbatcher is meant to make it easy to generate batches from Xarray datasets and feed them into machine learning libraries. As we wrote in its [roadmap](https://xbatcher.readthedocs.io/en/latest/roadmap.html), we are also considering...
### What is your issue? I plan to submit an abstract on xbatcher to the [13th Symposium on Advances in Modeling and Analysis Using Python](https://annual.ametsoc.org/index.cfm/2023/program-events/conferences-and-symposia/13th-symposium-on-advances-in-modeling-and-analysis-using-python/) at AMS 2023 under "New...
### What is your issue? As mentioned in https://github.com/pangeo-data/xbatcher/pull/71#pullrequestreview-1073348007, the xbatcher test suite needs improvements. This issue will track necessary improvements to the tests (additions are welcome): - [x] Test...
### What is your issue? We should update https://github.com/xarray-contrib/xbatcher/blob/main/xbatcher/__init__.py to use importlib.metadata rather than pkg_resources for accessing version information (ref. https://setuptools.pypa.io/en/latest/pkg_resources.html)
Add `BatchGenerator.to_zarr` and `BatchGenerator.from_zarr` to make it possible to save generated batches to zarr and later load them from zarr. By chunking along the batch dimension this enables fast data-loading...
I've recently been learning about [Hugging Face Datasets](https://huggingface.co/docs/datasets/index). It's a great data sharing platform for ML. The `datasets` package is based on tensorflow datasets. It would be great to think...
I've had a lot of luck using [sphinxcontrib-apidoc](https://pypi.org/project/sphinxcontrib-apidoc/) to automatically generate the API `.rst` files when the sphinx documentation is rendered and it has saved me a lot of time....
@jhamman just presented on some updates to xbatcher including the new data loader interfaces from #25. I tried to find a documented way of using it and I don't see...
In #37, @robintw wrote: > 2. How do you put batches back together after processing? > My machine learning model is producing a single value as an output, so for...