xarray-beam icon indicating copy to clipboard operation
xarray-beam copied to clipboard

Support opening datasets with file-like objects in a Beam pipeline

Open alxmrs opened this issue 4 years ago • 0 comments

I experimented a bit more with this based on @mjwillson's suggestion.

Amazingly, it seems that uses file-like objects in Xarray does actually work as used here, though making a local copy might still have better performance.

What doesn't work yet -- but hopefully with small upstream changes to Xarray could work -- is passing xarray datasets opened with these file-like objects into a Beam pipeilne. That could let us do the actual data loading from netCDF in separate workers, which could be quite a win!

Originally posted by @shoyer in https://github.com/google/xarray-beam/pull/31#discussion_r696246752

alxmrs avatar Sep 15 '21 05:09 alxmrs