xarray-beam

xarray-beam copied to clipboard

Consider omitting unchunked dimensions from Key objects created with DatasetToChunks

1

Currently we have (from https://xarray-beam.readthedocs.io/en/latest/read-write.html): ```python with beam.Pipeline() as p: p | xbeam.DatasetToChunks(ds, chunks={'time': 1000}) | beam.MapTuple(print_summary) ``` ``` Key(offsets={'lat': 0, 'lon': 0, 'time': 0}, vars=None) with Key(offsets={'lat': 0, 'lon':...

shoyer

Running pipelines on AWS?

1

Hello! Newbie here. :) Any tips for someone who would like to try an xarray-beam rechunking pipeline on AWS? (USGS is wedded to AWS at the moment)

rsignell-usgs

Adding support for `MergeDim`s and Split Variables to `FilePatternToChunks` transform.

3

Fixes #29 and #38.

alxmrs

For consistency with `DatasetToChunks`, consider calling this `chunks`. You might also add a `split_vars` parameter? _Originally posted by @shoyer in https://github.com/google/xarray-beam/pull/31#r713514917_

alxmrs

I experimented a bit more with this based on @mjwillson's suggestion. Amazingly, it seems that uses file-like objects in Xarray _does_ actually work as used here, though making a local...

alxmrs

Better documentation of behaviour with irregular source chunks

5

Hiya! In some source datasets the original chunks have irregular sizes. For example one netcdf file per year, where some years are leap years hence shorter than others. Although the...

mjwillson

Some notes on alignment with pangeo-forge-recipes around "keys"

1

At today's Pangeo Forge meeting, @alxmrs told us a bit more about Xarray Beam. I'm happy to note that we are aligning a bit around certain abstractions. I'm not proposing...

rabernat

alxmrs

These could facilitate directly opening data from Zarr using idiomatic patterns in Xarray-Beam (e.g., using Xarray's lazy indexing machinery instead of dask). I'm imaging `open_zarr()` returning a tuple of values...

shoyer

WIP: prototype of multi-stage rechunking

copybara-service[bot]